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About EURACT 


The European Academy of Teachers in General Practice and Family Medicine 
(EURACT) was launched in March, 1992. The overall aim of the Academy is 
“to foster and maintain high standards of care in European general practice by 
promoting general practice as a discipline by learning and teaching.” 

Since the launch of the Academy it has grown to be one of the largest per- 
sonal membership organisations in Europe, with over 700 members in nearly 40 
European countries. Teachers in each country in the WONCA European region 
may be members, and each country with at least three members has one repre- 
sentative on the Council, which is the ruling body. 

With the formation of the WONCA Region Europe/European Society of 
General Practice/Family Medicine, EURACT has become one of the network 
European organisations with a special interest in education. 

The EURACT website http://www.euract.eu hosts EURACT official docu- 
ments, together with specific guidelines, statements and other educational re- 
sources. Among these are the Dynamic Interactive Databases of Specialist Train- 
ing and Continuing Medical Education/Continuing Professional Development 
in General Practice/Family Medicine which provide up-to-date information 
about these topics in EURACT member countries. The website also offers a fo- 
rum and chat room and an “Ask the Expert” service exclusively for individual 
EURACT members. 

All teachers of general practice in the WONCA European region are invited 
to apply for membership of the Academy. Applications should be made to your 
national representative (see website for details). All applicants for full member- 
ship must be family doctors active in teaching general practice. 


www.euract.eu 
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1 Executive summary: 
Introduction to the EURACT Performance Agenda 


Stefan Wilm 


General practice/family medicine is a pivotal part of primary care systems: This 
fact is widely recognized all over the world!3, being popular with patients able 
to retain a personal relationship with their doctor in the increasingly impersonal 
world of health-care delivery, and with politicians because of its inherent cost 
effectiveness’. Evidence has shown that health care systems based on effective 
primary care provided by trained general practitioners working in the commu- 
nity are not only cost effective but provide effective care for patients”. This has 
been recognized in Europe, where, for the majority of European countries, gen- 
eral practice/family medicine is an important, fundamental part of the health 
care systems. The European Union has built general practice into its healthcare 
policies, and it is being developed by non-member states as well‘. General prac- 
titioners play a central role in ensuring that patients acquire appropriate health 
care provision. Although there has been a great deal of progress in the develop- 
ment of general practice in Europe, there remain large differences in the way this 
has been done in different countries and in the manner in which family doctors 
are integrated into the health care system’. 

A big step forward was taken in 2002 when WONCA Europe, the academic 
body for general practice, published the new European definition of general prac- 
tice/family medicine®. This work had come about as a result of revisiting previous 
definitions, as it had been felt that these were outdated and needed revising for 
the 21st century. At the outset it was recognised that the essential elements of 
the discipline of general practice/family medicine needed to be defined first, and 
only when this had been done could the role definition of the family doctor be 
derived from them. 

The definition contains the eleven characteristics which are fundamental to 
the discipline and are generalisable to all health-care systems in all countries 
regardless of contextual differences. These were then combined into a role de- 
scription of the family doctor. 

The document does not stop there, but goes on to explore the competencies 
that are required to become a skilled exponent of the discipline. It describes 
how these characteristics can be grouped together into six core competencies, 


and how the acquisition of these can be converted into abilities every family 
doctor should be able to master, and eventually into actual performance’ (see 
chapter 2). 

EURACT developed this thinking further and produced its educational 
agenda in 2005. EURACT, the European Academy of Teachers in General Prac- 
tice/Family Medicine, has the constitutional aim to foster and maintain high 
standards of care in European general practice/family medicine by promoting 
the discipline through learning and teaching. The EURACT Educational Agen- 
da" provided a comprehensive framework to teach the core competencies being 
a blueprint for a common curriculum. The educational agenda is a longitudinal 
strategy in regard to the learning process, including basic medical education 
(BME), specialty or vocational training (ST), and continuing medical education 
(CME)/continuing professional development (CPD), but focussing on specialty 
training. Based on learning in a primary care setting, it sets learning aims, is 
based on modern educational methods and monitors progress of learners as well 
as the achievement of goals. It is developed for educators, curriculum builders, 
educational boards, policy makers and other medical specialists. 

Just like the European definition, the educational agenda is supposed to 
be applicable regardless of differences in health care systems, educational pro- 
grammes and health care policies. It aims at harmonization of learning outcomes 
of educational programmes in general practice/family medicine in all European 
countries. 

General practice/family medicine is an academic and scientific discipline 
with its own educational content, research, evidence base and clinical activity, 
and a clinical specialty orientated to primary care. The discipline has a contribu- 
tion to make to the education of doctors at all stages in their training. Changes in 
attitudes of medical learners generally occur after learners have the opportunity 
to grapple with primary care problems themselves and observe their teachers 
coping successfully with problems presented". 

At the BME level, all students", irrespective of their future professional ca- 
reer choice", should follow a curriculum, which is directed at enabling them 
to understand the role and specificity of primary care and general practice'^^. 
Early clinical exposure of medical learners for brief periods from several days 
to weeks at the beginning of medical school can lay the groundwork for the 
medical student to help make sense of all medical training’. Lectures or small 
group seminars in the pre-clinical period can be effective in providing some 
of the knowledge required before starting work with patients". The prime peri- 
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Does Performance 


Shows how Competence 


Knows how 


Figure 1: Miller's pyramid 


od for training learners in primary care management is the student clerkship/ 
attachment/rotation in general practice, lasting several weeks occurring in the 
later years of medical school!*””. 

The in-depth preparation of medical graduates to gain the competencies of 
family physicians takes place at the ST level. This part of medical education usu- 
ally lasting several years, and in the EU a minimum of three years”, allows train- 
ees to acquire the knowledge, skills and attitudes which are essential for safe and 
effective professional general practice. A large part of the programme will be 
based on teaching practices”). 

The CME/CPD level comprises the longest phase of professional education 
(life-long learning). In CME, graduated general practitioners continue to learn 
and change in practice in a lifelong learning scheme, thus optimizing their ac- 
tual performance. CPD is understood wider as a process of planned and indi- 
vidually tailored learning in practice with a focus on the quality of care?. CPD 
may include the identification of individual learning needs, construction of a 
learning agenda, drawing a concrete personal learning plan, and controlling this 
in an educational portfolio format. 

Following Miller's terminology of assessment? (figure 1), the EURACT Ed- 
ucational Agenda and EUPA define competence as the capability to successful- 
ly perform discrete observational tasks in a defined assessment environment, 
in isolation from actual work. It includes the level of “knowing” (basic facts), 
"knowing how" (able to apply knowledge) and "showing how" (able to show 
skills) but it excludes the "doing" level, the performance in practice. Competen- 
Cy-based assessments measure what doctors do under examination conditions". 
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Performance is defined as the level of actual performance in clinical care and 
communication with patients in daily practice. It relates in the Miller terminol- 
ogy to the “doing” level. It is considered highly dependent on existing healthcare 
conditions and requirements, financial and structural opportunities, practice 
opportunities and support. Performance-based assessments measure what doc- 
tors do in practice. 

Rethans and colleagues? argue that Miller triangle implicitly assumes that 
competence predicts performance, but that the exact relationship between com- 
petence and performance is much more complicated. Factors such as time pres- 
sure, day of the week, mood of patient and doctor or impact of the fore-going 
examination influence clinical performance just as does individual competence 
in specific domain-related knowledge areas. System-related influences include 
government programmes and initiatives, patient expectations, guidelines or pol- 
icies developed by the practice facility, time, and accessibility to other health 
professionals. Individual-related influences cover the physical and mental health 
of the GP, their state of mind at the time of the performance assessment, and 
their relationship with others, including patients, peers, and their own family”. 
Therefore, contemporary assessment demands a more flexible, interactive figure 
than a pyramid, allowing a view of the performance of an individual doctor from 
different angels. Rethans et al. have suggested the inversion of Miller’s pyramid, 
focussing on the top two tiers (figure 2): 

Clearly, competence is an important prerequisite for performance, and is 
therefore represented in the triangle in the centre. But at least two further tri- 
angles (system-related and individual-related influences) carry performance on 
their shoulders, identifying performance as a product of many aspects. EUPA 
follows this complex model of performance. While it is quite reasonable to seek 
an association between competence and performance, it is unreasonable to ex- 
pect it to be strong”. 

The future of the discipline of general practice/family medicine depends 
on many factors”, but certainly these include the quality of actual daily perfor- 
mance of general practitioners in this field. The EURACT Educational Agenda?! 
has provided a comprehensive framework to teach the six core competencies 
outlined in the European definition in BME and ST and to monitor and assess 
the achievement of goals at the end of training. The Educational Agenda defines 
a well performing GP as a 
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System related 
influences eg. 
practice facilities, 


Individual related 
influences eg. 
health 


Figure 2: The Cambridge Model for delineating performance and competence?’ 


professional which combines, uses and implements different elements of the pro- 
fessional skills in a fluid, normal and interiorized manner, knowing at every mo- 
ment what are the background options and the key features related to it. 


Now the questions remain: In actual practice, after gaining the competen- 
cies that are required to become a skilled exponent of the discipline and after 
finishing ST, 

e Do general practitioners perform proper in regard to the six core compe- 
tencies? 

e Where, how and by whom can this be assessed? 

e How and from whom can general practitioners get feedback on the identifi- 
cation of learning needs and on optimizing their daily performance? 


To provide answers to these questions, EURACT has produced the present 
EURACT Performance Agenda (EUPA), being the third paper in a row follow- 
ing the European Definition of General Practice/Family Medicine by WONCA 
Europe in 2002, and the EURACT Educational Agenda in 2005. This document 
will close the loop between teaching knowledge, allowing students and trainees 
to gain competencies, and assessing actual performance of GPs in daily practice. 

The first idea of EUPA was born in EURACT Council in 2007. Following an 
international study performed in Canada, Australia, New Zealand and the UK”, 
a task force conducted a survey asking all Council delegates from 38 nations 
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about performance agendas or official documents in specialty training or con- 
tinuing professional development in their countries. Council delegates from 34 
countries responded. It was found that only a very few countries (e. g. UK, Israel) 
had a comparable document or policy on performance assessment in general 
practice/family medicine that went beyond a simple list of competencies. Af- 
ter a review of the literature, discussion in small groups involving all EURACT 
Council members took place over two years, following the chapters of the Ed- 
ucational Agenda and using the same terminology. Design of the documents 
outline, compilation of the final draft of the chapters focussing on the six core 
competencies and theoretical framing was done by a small group of authors. 
The final version of the document was approved by the EURACT Council in 
October 2013. 

EUPA is a general, uniform and basic agenda of performance elements 
grouped in the six core competencies every general practitioner has to master 
in daily practice. Like the other two documents, it is supposed to be applicable 
and adaptable to different countries with different systems instead of being too 
normative. It deals with the process and result of actual work in daily practice, 
not with a teaching/learning situation, thus focussing on the CPD level, the 
longest phase of professional education (see chapter 3). It is concentrating on 
measurable/observable/assessable performance in general practice, applicable 
to various tasks (e.g. communication with patients, clinical routines, practice 
management etc) and in a wide range of settings. Thus, EUPA is not a list/di- 
rectory outlining which competencies, skills and procedures are mandatory for 
general practice and which ones are optional. 

Chapter 2 provides an overview of the six core competencies. Chapters 5 to 
10 focus on each of the six. In every chapter, case vignettes of abilities in doctors’ 
daily practice illustrate performance elements, their adequate assessment meth- 
ods and the role of the assessor (tables 5-20). 

The goal of such assessments is to see the whole picture of a practitioner's 
performance. A varied palette of methods is necessary to achieve this goal?. In 
general, assessment of a practitioner's performance can only be done on-site, at 
his/her workplace (workplace-based assessment; see chapter 11), preferably by 
direct observation. Performance assessment should refer to a theoretical frame- 
work (see chapter 4), should be organised in a programmatic way, i. e. embedded 
in daily practice routines, and cannot be improvised or organised as something 
"external" to continuing professional development (CPD) activities?^?. Ensur- 
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ing the competent clinical performance of practicing doctors on an ongoing ba- 
sis presents many challenges. The public, its elected representatives, employers, 
and doctors themselves may all have expectations and anxieties that need to 
be reconciled***’. Because of the severe consequences an assessment of practice 
performance may have, it is essential that the procedure is both defensible to the 
stakeholders and fair in that it distinguishes well between good performers and 
underperformers*. Preferably, these programmes should remain in the hands of 
the profession, and the role of assessment should be a production tool of mean- 
ingful feedback to the individual GP. 

The relation of performance assessment to quality management is discussed 
in chapter 3. The theoretical framework of the psychometrics and edumetrics of 
performance assessment is outlined in chapter 4. 

A systematic review has observed six different suitable methods of evaluating 
performance: simulated patients (unannounced visits”); video observation; di- 
rect observation; on-site peer assessment; audit/review of medical records; and 
portfolio or appraisal. Evidence supporting improvement in routine practice is 
lacking so far“. Other authors have suggested other methods, e. g. feedback from 
patients, feedback from relatives of patients, and feedback from staff, or the use 
of checklists}? 

Chapter 11 (glossary) provides more information about the above key per- 
formance assessment methods and defines other terms. 

EUPA can help to shape various performance assessment activities held lo- 
cally in general practice/family medicine, such as: 


e Continuing professional development (CPD) cycles; 

e  Reccertification/re-accreditation/re-validation/re-licensing procedures; 
e Peer hospitation programmes (e.g. visitatie programme”); 

e Practice audit programmes in quality management 


and can give orientation for self assessment for reflective practitioners in their 
CPD. 

Itis hoped that the EURACT Performance Agenda EUPA will encourage Eu- 
ropean general practitioners to initialize performance agendas adapted to their 
national health system to further strengthen the role of general practice/family 
medicine in their country. 
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2 Overview of the six core competencies 


The present EURACT Performance Agenda (EUPA) is the third paper in a row 
following the European Definition of General Practice/Family Medicine (WON- 
CA Europe) in 2002*, and the EURACT Educational Agenda in 2005” provid- 
ing a framework for teaching and assessment. 

The EURACT Educational Agenda is derived from the European definition, 
which defined the eleven characteristics of the discipline, and from them derived 
six core competencies of family doctors and three essential application features. 
For each of the six core competencies the EURACT Educational Agenda defines 
learning objectives, teaching and assessment methods, and makes some notes on 
time and setting in the curriculum (see chapter 1). The EURACT Performance 
Agenda (EUPA) focuses on methods and settings to assess the actual perfor- 
mance in the area of the six core competencies (tables 5-20 in the following 
chapters) to see the whole picture of a practitioner’s performance in daily prac- 
tice. 

Defining the discipline of general practice/family medicine and the specialist 
family doctor leads directly to the core competencies of the general practitioner/ 
family doctor. These competencies should be taught to the basic doctor after 
BME, and should be maintained as part of lifelong professional performance. 
For that reason the six core competencies are reflected in the EURACT Educa- 
tional Agenda as well as in the related EURACT Performance Agenda (EUPA). 

Core means essential to the discipline, irrespective of the health care system 
in which they are applied. In the European Definition they are grouped in three 
paragraphs, in relation to the discipline ($1), to the practice of the specialty ($2) 
and some basic features as background (§3). 


§1 


The eleven central characteristics that define the discipline relate to eleven abili- 

ties that every specialist family doctor should master. They can be clustered into 

six core competencies. Each cluster is described by their main aspects: 

1. Primary care management includes the ability: 

e To manage primary contact with patients, dealing with unselected prob- 
lems; 

e To cover the full range of health conditions; 
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To co-ordinate care with other professionals in primary care and with other 
specialists; 

To master effective and appropriate care provision and health service util- 
isation; 

To make available to the patient the appropriate services within the health 
care system; 

To act as advocate for the patient. 


Person-centred care includes the ability: 

To adopt a person-centred approach in dealing with patients and problems 
in the context of patient’s circumstances; 

To apply the general practice consultation to bring about an effective doc- 
tor-patient relationship, with respect for the patient's autonomy; 

To communicate, set priorities and act in partnership; 

To provide longitudinal continuity of care as determined by the needs of the 
patient, referring to continuing and co-ordinated care management. 


Specific problem solving skills include the ability: 

To relate specific decision making processes to the prevalence and inci- 
dence of illness in the community; 

To selectively gather and interpret information from history-taking, physi- 
cal examination, and investigations and apply it to an appropriate manage- 
ment plan in collaboration with the patient; 

To adopt appropriate working principles, e.g. incremental investigation, us- 
ing time as a tool and to tolerate uncertainty; 

To intervene urgently when necessary; 

To manage conditions which may present early and in an undifferentiated 
way; 

To make effective and efficient use of diagnostic and therapeutic interven- 
tions. 


Comprehensive approach includes the ability: 

To manage simultaneously multiple complaints and pathologies, both acute 
and chronic health problems in the individual; 

To promote health and well being by applying health promotion and disease 
prevention strategies appropriately; 

To manage and co-ordinate health promotion, prevention, cure, care and 
palliation and rehabilitation. 


§2 


Community orientation includes the ability: 
To reconcile the health needs of individual patients and the health needs of 
the community in which they live in balance with available resources. 


Holistic modelling includes the ability: 
To use a bio-psycho-social model taking into account cultural and existen- 
tial dimensions. 


To practice the specialty, the competent practitioner implements these compe- 
tencies in three important areas: 


a. 


e 


$3 


In the daily clinical tasks: 

Manage the broad field of complaints, problems and diseases as they are 
presented; 

Master long-term management and follow-up; 

Balance evidence and experience in an effective way. 


In the communication with patients: 

Structure the consultation properly; 

Provide information that is easily understood and to explain procedures 
and findings; 

Deal adequately with different emotions. 


In the management of the practice: 

Provide appropriate accessibility and availability to the patients; 

Organise, equip and financially manage the practice, and collaborate with 
the practice team; 

Cooperate with other primary care staff and with other specialists. 


As a person-centred scientific discipline, three background features should be 
considered as fundamental: 


a. 


Contextual: 

Use the context of the person, the family, the community and their culture 
in diagnosis, decision making and management planning; 

Show personal interest in the patient and his environment and be aware 
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of the possible consequences of disease for family members and the wider 
environment (including working environment) of the patient. 


Attitudinal: 

Based on the awareness of one’s own capabilities and values; 

Identifying ethical aspects of clinical practice (prevention/diagnostics/ 
therapy/factors influencing lifestyles); 

Justifying and clarifying personal ethics; 

Being aware of the mutual interaction of work and private life and striving 
for a good balance between them. 


Scientific: 

Being familiar with the general principles, methods, concepts of scientific 
research, and the fundamentals of statistics (incidence, prevalence, predict- 
ed value etc.); 

Having a thorough knowledge of the scientific backgrounds of pathology, 
symptoms and diagnosis, therapy and prognosis, epidemiology, decision 
theory, theories of the forming of hypotheses and problem-solving, preven- 
tive health care; 

Being able to access, read and assess medical literature critically; 

Adopting a critical and research based approach to practice and maintain- 
ing this through continuing learning and quality improvement. 


The interrelation of core competencies, implementation areas and funda- 


mental features characterises the discipline and underlines the complexity of 
the specialty. 


It is this complex interrelationship of core competencies that guides and is 


reflected in EURACT’s Educational Agenda and Performance Agenda. 
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3 The relation of performance assessment 
to quality management 


Janko Kersnik, Zalika Klemenc-Ketis 


The definition of performance in the present EURACT Performance Agenda is 
the level of actual performance of physicians in clinical care and communication 
with patients in daily practice. Quality management is the evaluation of services 
provided and the results achieved as compared to accepted standards. The re- 
sults of health care are compared with standard results and the deficiencies or 
problems defined serve to plan and introduce improvement actions. In quality 
management, this is called a PDCA cycle which has four steps: 

e Plan what we are going to do. In this step we assess where we are, where we 
need to be, why this is important, and plan how to close the gap. Identify 
some potential solutions. 

e Dotry out or test the solutions (sometimes at a pilot level). 

e Check to see if the countermeasures you tried out had the effect you hoped 
for, and make sure that there are no negative consequences associated with 
them. Assess if you have accomplished your objective. 

e Act on what you have learned. If you have accomplished your objective, put 
controls into place so that the issue never comes back again. If you have not 
accomplished your objective, go through the cycle again, starting with the 
planning step. 


3.1 Relation of performance assessment to quality assessment 


The aim of any assessment in quality management is to see the level of quality 
of care (i.e. structure, processes and outcomes of care) in real practice settings. 
This serves as an indicator of quality level (used e.g. for ranking, certification, 
accreditation) and/or as a starting point to initiate necessary changes, which 
should lead to quality improvement, and as a benchmark to be compared with 
the assessment after change. 

Performance assessment is therefore a core procedure of any quality assess- 
ment. However, any assessment in quality management has some inherent ten- 
dency to change behaviour, practice, and structure in order to get better out- 
comes. On the other hand, assessment in the educational realm is strictly for 
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educational purposes in case of formative assessment, or for certification pur- 
poses in the case of summative assessment. 

In quality management, there is no other way to assess quality of care than by 
some direct or surrogate way of observation of structure, process and outcomes. 
In education, knowledge, skills and attitudes as well as competence (see chapter 
1) can be assessed by a variety of methods, which yield more or less reliable re- 
sults with a good supposition that certified doctors will show good performance. 
Actual performance can be assessed during or at the end of the educational pro- 
cess and is best assessed in physicians’ everyday practice settings. However, this 
is not considered as quality assessment as it is used solely for educational (CPD) 
or certification purposes. But when performance is assessed with the aim of im- 
proving the quality of physicians’ work as a part of quality improvement (i.e. re- 
peatedly in different stages of the PDCA cycle), it becomes a method for quality 
assessment and/or improvement (see also“). 
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4 The psychometrics and edumetrics 
of performance assessment 


Jean-Marie Degryse, Valéry Dory, Cees P. M. Van der Vleuten 


4.1 Introduction: 
performance assessment - climbing Miller’s pyramid 


For over more than twenty years now, Millers pyramid? has been used as a 
framework to define different levels of assessment of medical competence (see 
chapter 1). Historically, emphasis has been placed on assessment at the lower 
layers, directed at knowledge, application of knowledge and demonstration of 
skills. More recent developments are concentrated at the top: the “does” level. 
Performance assessment is predominantly assessment in the workplace. Work- 
place-based assessment™ is likely to become an essential part of both licensure 
and re-certification procedures in family medicine. Figure 3 offers an overview 
of the different assessment methods that are commonly used at different levels 
of Miller’s pyramid. 


Stimulus format: habitual practice performance 
Response format: direct observation, checklists, rating scales, 
narratives 


Does 


Stimulus format: hands-on (patient) standardized scenario 
or simulation 
Response format: direct observation, checklists, rating scales 


Shows how 


Stimulus format: (patient) scenario, simulation 
Response format: menu, written, open, oral, com- 
puter-based 


Knows how 


Stimulus format: fact-oriented 
Response format: menu, written, open, com- 
puter-based, oral 


Figure 3: Miller’s pyramid showing different levels of assessment and the associat- 
ed assessment formats 


From the psychometric perspective, workplace-based assessment offers new 
challenges. Some researchers point to the threats to reliability and validity from 
uncontrollable variables, such as patient mix, case difficulty and patient num- 
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bers’. Others show that the utility of performance assessment results is compro- 
mised by lower inter-rater reliability and rater effects such as halo, leniency or 
range restriction’. As a consequence attempts to improve performance assess- 
ment typically focus on standardization and objectivity of measurement by ad- 
justing rating scale formats and eliminating rater errors through rater training”. 

Traditional psychometric approaches towards assessment tend to focus ex- 
clusively on quantitative properties of assessment outcomes. However in this 
chapter we will advocate that this rigorous psychometric approach might limit 
more meaningful approaches to performance assessment. Performance assess- 
ment is essentially a judgment and decision making process in which rating out- 
comes are influenced by interactions between individuals and the social context 
in which assessment occurs. Also, depending on the “stakes” of the performance 
assessment, the focus as well as the design of the procedure will be different. In 
a “low stake” context, emphasis will be put on the production of rich and mean- 
ingful (multi-source) feedback in order to support and enhance learning. While 
in a “high stake” context the focus will shift to the gathering and aggregation 
of information from as many measurements as possible in order to produce a 
robust final judgment and decision as an alternative to a single “final score”. Both 
designs require viewing of the individual doctor’s performance from different 
angels as visualized in the inverted Miller's triangle, the Cambridge Model (see 
figure 2 in chapter 1). 

Performance assessment in daily practice needs a theoretical framework 
to be scientifically sound. Therefore, the practical chapters 5-10 of EUPA are 
flanked by this theoretical chapter about the psychometrics and edumetrics of 
performance assessment. We will first provide an overview of the lessons that 
can be derived from the assessment literature and three decades of research. 
They have been synthesized in a paper by Van der Vleuten et al? and relate in 
particular to assessment of the three first layers of Miller's pyramid. We will then 
confront those with the typical demands and constrains of workplace-based as- 
sessment and present some of the solutions and alternative approaches that have 
been proposed to assess medical competence at the top layer. 


i We use Workplace-Based Assessment here as a term that relates to performance assess- 
ment in a “real life” professional environment (see chapter 11 - Glossary). 
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4.2 Lessons learned from three decades of research 


4.2.1 Competence is specific, not generic: Content specificity 
as a central issue in assessment of medical competence 


An issue that dominates (bedevils) every effort to develop a reliable assessment 
procedure at any level of Miller’s pyramid is the problem of “content-specificity”. 
In assessment of medical competence, more than in any other field, the scores on 
a test (or a case, or an item) that was designed to assess a specific content domain 
predict scores on another domain very poorly. This variability of performance 
of candidates across task appears to be one of the most consistent findings in 
all measurements of clinical competence‘. A direct consequence of this con- 
tent-specificity is that assessment blueprints must ensure broad sampling across 
a huge variety of task in order to achieve sufficient reliability. Or in other words: 
tests containing a small sample of items (stations, observations, tasks, patient 
encounters) produce unstable or unreliable scores. Obviously this will also vary 
with the size of the domain being assessed, but even in smaller content domains 
the required sample size of test items/cases is usually high. In this respect some 
testing methods that allow broad sampling in less time might be considered as 
more reliable per unit of testing time than tests requiring more time per item. 
In order to produce adequate reliability coefficients within a high stake con- 
text, one should take into account that even efficient tests usually require several 
hours of testing time”. 

Research on a variety of assessment methods has consistently revealed that 
the most important determinant of score reliability is test length as is illustrated 
from the overview in table 1. Other determinants of variability that challenge the 
reliability of assessments, such as rater, patient or observer variability, are usually 
either less important or easier to control. 

A second practical consequence of this content specificity phenomenon is 
that when limited resources are available, the design of the assessment proce- 
dure should preferentially be adapted so that broad sampling of the domain that 
will be tested is guaranteed and a maximum number of cases/items is intro- 
duced. For instance when a limited number of observers are available as rat- 
ers of performance in stations in an Objective Structured Clinical Examination 
(OSCE), it is preferable to spread those over double as many stations instead of 
appointing them in pairs to half the number of stations®. 
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Case- Practice 


Testing RT i : 
5 : Based Oral Long Mini Video Incogni- 
tb 1 5 
pues A Short EME Exam? Case’ CE CEX* Assess-  toSPs? 
Essay? ment’ 
1 0.62 0.68 0.36 0.50 0.60 0.47 0,73 0.62 0.61 
2 0.76 0.73 0.53 0.69 0.75 0.64 0.84 0.76 0.76 
4 0.93 0.84 0.69 0.82 0.86 0.78 0.92 0.93 0.92 
8 0.93 0.82 0.82 0.90 0.90 0.88 0.96 0.93 0.93 
!Norcini et al., 1985 4Wass et al., 2001 7Ram et al., 1999 
?Stalenhoef-Halling et al., 1990 Petrusa, 2002 8Gorter, 2002 
Swanson, 1987 5Norcini et al., 1999 


Table 1: The reliability as a function of testing time (Van der Vleuten et al 2005°") 


If observational test methods are used (as in level 3 or 4 of Miller’s pyramid), 
other secondary factors might improve reliability. Standardization of the content 
of the assessment, as well as training of the observers, might contribute to a fur- 
ther improvement of the reliability. 

A final issue that should be addressed here is that “reliability is relative” i.e. 
the reliability of a measure is intimately linked to the population to which one 
wants to apply the measure. There is literally no such thing as the reliability of a 
test. A reliability coefficient has meaning only when applied to a specific popu- 
lation. Reliability is not a fixed characteristic of an assessment method but will 
depend upon the “true variability” of the scores in a population. 

In order to understand this statement, we should consider the way reliability 
is conceptualized in classical test theory. Reliability is defined as the relationship 
between the “true variance of the scores” to the observed variance of test scores, 
the latter being defined as the sum of the true variance and a variance linked to 
measurement error. In a formal way this can be expressed as: 


2 2 
on 


TE [o 
reliability = —5 " 5 
o o, +o, 


Thus the reliability coefficient expresses the proportion of the total variance 
in the measurements (c,’ +ø, ) that is due to “true differences between sub- 
jects” (o, ). If o, is zero, the reliability coefficient will be 1. Assuming that o,” 
remains constant, higher o,” will produce higher reliability coefficients. 
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4.2.2 Objectivity is not the same as reliability 


Reliability does not co-vary with the objectivity of methods. The so called “sub- 
jective” tests can be reliable and objective tests can be unreliable, all depending 
on the sampling and the method”. As an example we can refer to the histori- 
cal development of the Objective Clinical Examination format. The OSCE was 
originally designed in order to overcome the problem of subjectivity that was 
present in classical clinical exams. The solution was sought in objectivity and 
standardization of the score forms and cases and in training of the observers. 
However the reliability of the OSCE turned out to be as dependent of the broad 
sampling of its content (i.e. the stations) as any other method”. Table 2 shows 
the effect of test length (expressed as number of stations) on the reliability of a 
typical OSCE that is part of a national licensing examination of family physi- 
cians in Flanders. 


Reliability index 


Test length (minutes) Number of stations Norm referenced Domain referenced 
80 10 .60 ‚55 
120 15 .69 .64 
160 20 .75 .71 
200 25 .79 15 
240 30 .82 18 
280 35 84 81 
320 40 .86 .83 


Table 2: Reliability indices of the OSCE as a function of testing time (Degryse 2003) 


Another interesting finding from research on the OSCE is the strong correla- 
tion that was found between global rating scales and checklist ratings”. Global 
ratings are associated with a slight decrease in inter rater reliability, but this was 
offset by a larger gain in inter station reliability. 


Compared with the more analytical checklist scores, global holistic judgments 
tend to pick up on elements in candidates' performance which were more gener- 
alisable across stations. This is a clear and intriguing first indication that human 
expert judgment could add (perhaps even incrementally) a meaningful "signal" 
to measurement instead of “noise”. 
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Another issue that should be addressed here is that reliability is not neces- 
sarily related to agreement between judges/raters and in some cases it can even 
be inversely related to it! The scores produced by different raters in an observa- 
tional test can vary consistently but still generate reliable mean scores. On the 
other hand, if all candidates on all occasions are rated above average, the agree- 
ment among raters is perfect but the reliability by definition is zero. What ac- 
tually happens is an intriguing and somewhat paradoxical interaction between 
the problem of content specificity and assessor variability. The global nature of 
the judgment may be more subjective, but it dampens or attenuates the content 
specificity problem and therefore leads to improved reliability overall. 


4.2.3 Compromising and making informed choices 


Every assessment procedure is a by definition a compromise. The selection of a 
particular assessment method may involve factors that are not of a psychomet- 
ric nature. Trade-offs between what is desirable and achievable or feasible are 
inevitable. 

In order to clarify the compromises involved, Van der Vleuten et al.°' have 
presented some years ago a model to define the utility of an assessment method 
(figure 4). Reliability (R), Validity (V) and educational impact (E) should be 
part of the model for obvious reasons". However, in educational practice the 
choice of a particular assessment method will often be influenced by other con- 
siderations such as opinions, sentiments and traditions of teachers, students and 
institutions. That is why the model introduces two additional variables: the ac- 
ceptability (A) and the cost (C) and logistic burden induced by the procedure. 
The utility of an assessment method is represented by an equation in which the 
relationship among variables is deliberately conceived as multiplicative. If one of 
the elements is zero, the utility will be zero. 

In practice, one will always be required to compromise and assign different 
weights in different individual situations, depending on the context and pur- 
pose of the assessment. In situations where the assessment involves a high-stake 
examination with decisions having marked consequences on the future of ex- 


* It should be noted that from a psychometric perspective reliability and validity are not 
independent variables. Reliability is a prerequisite for validity. The reliability determines 
the maximum validity that can be achieved. The relation between reliability and validity 
can be expressed mathematically as follows: r,, = du where r, stands for reliability and 
r „for the validity of a test. 
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aminees, reliability will probably bear a heavier weight. While in the context of 
in-training assessment, where the final decision is based on many assessments, 
one would probably be prepared to compromise more on reliability in favour of 
the educational impact of the assessment“. In other situations a higher weight 
should be attributed to the potential educational impact of a procedure even if 
one has to compromise on reliability. For instance in a CPD programme, a less 
reliable test procedure might be preferred due to the inherent signal it sends out 
which guides life-long learning of practitioners on a micro and macro level. In 
this context it is essential that the assessment procedure is at its most transparent 
in order to allow the communication of information and an optimal steering 
effect. 


U = WR X wV X wE X wAX C 


U = Utility R = Reliability V = Validity 
E = Educational impact A = Acceptability C = Cost 
W = Weight 


Figure 4: The utility function of an assessment procedure 


4.2.4 What is being measured is determined more by 
the stimulus format than by the response format 


It has often been assumed that different assessment methods measure different 
aspects of medical competence. Summative assessment procedures and/or li- 
censing exams have often purposefully been designed as a battery of tests with 
different formats. The underlying assumption has been for instance that Multi- 
ple Choice Question (MCQ) tests would focus on knowledge, OSCEs on clinical 
skills, and more sophisticated written tests or computer based tests on elements 
of clinical reasoning. 

However, research has revealed that what is being measured is determined 
more by the stimulus format than by the response format. Provided the scores 
are reliable (which can be ensured by broad sampling), scores obtained on tests 
using different formats generally correlate with each other quite well. Cognitive 
activities follow the task that is posed in the stimulus format. A well designed 
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written knowledge test with rich contextual clinical vignettes can measure much 
more than factual knowledge, and conversely a poorly designed OSCE can tar- 
get only rudimentary, decontextualized, and technical skills. Validity - what is 
being measured - is not so much determined by the response format as by the 
stimulus format™. 

A practical consequence is that test developers should worry more about 
designing appropriate stimulus formats than about appropriate response for- 
mats. In developing stimuli, authenticity is essential also, provided the stimu- 
lus is tuned to the appropriate level of complexity. The first OSCEs consisted 
of short stations assessing clinical skills in a fragmented manner (e.g. station 1 
examination of the shoulder, station 2: abdominal examination, station 3: com- 
munication), which may be defensible at early stages of training. But, at a more 
advanced stage of training, integrated skills assessment is obviously a more ap- 
propriate stimulus format, since it provides a closer approximation of real clin- 
ical encounters. 


4.2.5 Assessment drives learning 


The impact of assessment, from selection to certification, on learning is signif- 
icant?. Before assessment even occurs, it influences learning by providing an 
external motivation and also by providing learners with cues which they will 
interpret in order to conduct and regulate their learning“. Learners are generally 
strategic in their efforts. They will allocate time and select learning and regula- 
tion strategies according to their understanding of the task at hand. This can lead 
to negative effects, e.g. learners using surface strategies such as rote learning to 
prepare for MCQs, or indeed memorizing OSCE checklist items. This can lead 
to a trivialization of the learning endeavour and counteract the effects of other 
components of the curriculum‘. 

During a test, learners are provided with an opportunity to practice a task 
which in itself can lead to improved performance. This type of impact is referred 
to as the testing effect and has been demonstrated for both tests of knowledge 
and tests of skills®”. 

Assessment also has a significant impact by providing feedback to learners, 
what Norcini et al refer to as the catalytic effect of assessment”. Feedback has 
been shown to be conducive to learning providing a certain number of condi- 
tions are met: the feedback should be timely, specific, based on observation, pro- 


30 


vided by a credible person, and qualitative rather than numerical??7*, Indeed 
feedback is not necessarily assimilated directly by learners. A study on general 
practitioners receiving multisource feedback found that assimilation depended 
on a process of reflection which balanced external feedback with evaluations 
based on self-assessment”. This process was also influenced by the emotional 
reaction of learners to feedback. Negative feedback that was inconsistent with 
an individual's self-assessment was more likely to be discounted. The authors 
suggest that this reflective process could be facilitated by a coach: interpreting 
feedback and using these interpretations to steer learning should be viewed as a 
complex sense-making process influenced by social interactions and the learn- 
ing climate. 

In view of the significant impact of assessment on learning, several authors 
have called for a shift from assessment of learning to assessment for learning, in 
which assessment is viewed as a key component of the curriculum and is used 
strategically to foster desirable learning behaviours”. 


4.3 Assessing competence at the “does” level 


Any assessment at the top level of Miller’s pyramid in an authentic context, will 
rely to a large extent on “expert judgment”. The term expert should be inter- 
preted broadly to include peers, supervisors, co-workers, teachers, and anyone 
knowledgeable about the work of the trainee/doctor, perhaps even the learner 
him or herself. The assessment consists of gathering these judgments in some 
quantitative or qualitative form. As with OSCEs the dominant response format 
is some form of observation structure (numerical rating scale, scoring rubrics, 
free text boxes) on which a judgment is based. Unlike the OSCE, however, the 
stimulus format is the authentic context, which is essentially unstandardized 
and relatively unstructured. In addition to scoring performance on rating scales, 
assessors are often invited to write narrative comments about the strengths and 
weaknesses of a learner performance. Roughly sketched two types of assess- 
ment instruments have been used. The first involves judgment of performance 
based directly on observation or on the assessor’s exposure to the learner’s per- 
formance. The second consists of aggregation instruments that compile infor- 
mation from multiple sources over time" (table 3). 
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Direct performance measures 


* Individual encounter methods Assessment is confined to a single concrete situation 
e.g. Mini-clinical-evaluation exercise (Mini-Cex) 

Direct observation of practice skills (DOPS) 

Professionalism mini-evaluation (P-Mex) 

Video-observation of clinical encounters 
* Long-term methods Performance is assessed over a longer period of time 
e.g. Peer assessment 

Multisource or 360 ? feedback (MSF) 


Aggregation methods 


e.g. Logbook Continuous sampling of performance 
Portfolio 


Table 3: Overview of methods used to assess medical competence at the "does" level 


4.3.1 A sample is required to achieve reliable inferences 


All modern methods of assessment at the “does” level allow for frequent sam- 
pling across educational or clinical contexts and across assessors. The need to 
deal with content specificity (see above) means that sampling across a range 
of context remains invariably important”. The subjectivity of expert judgments 
needs to be counterbalanced by additional sampling across assessors in order 
to produce "aggregated" information. Rather than targeting the production of a 
"reliable" final score, the final aim of any assessment at the “does level" will be to 
produce a robust judgment. 


4.3.2 Bias in expert judgment 


Global judgments are prone to bias, probably much more than structured ana- 
lytical methods. Two examples: with direct observations, inflation of scores has 
often been noted”, and in multisource feedback, careless selection of assessors 
can induce important bias®. The context in which the assessment takes place is 
another important potential source of bias in particular when the assessor(s) 
have an educational relationship with the trainee or junior doctor. They might 
be tempted to inflate scores to avoid negative impacts on learners, the relation- 
ship or themselves (e. g. having to justify failing a learner)*'. One solution to re- 
duce this type of bias could be to remove the summative aspect of the assessment 
from the individual encounter. The tutor/assessor should not assess whether the 
learner is a good doctor, but concentrate on what happens in a specific encoun- 
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ter in order to produce meaningful feedback. Summative decisions (pass/fail) 
should be based on multiple sources of assessment within and across methods 
and should preferentially involve external assessors. As mentioned earlier: the 
robustness of the final decision will rely on the meaningful aggregation of differ- 
ent sources of information. 

There is a growing interest in understanding rater cognition. Initial studies 
suggest that in the same way as doctors use illness scripts and examples of ac- 
tual patients to diagnose diseases, raters use mental models or scripts of learner 
performance and perhaps exemplars of prior learners to gauge learner perfor- 
mance?535555 This may explain the somewhat idiosyncratic nature of judgments 
based on personal experience with learners. Training may help communities 
of raters develop shared mental models and improve the consistency of ratings 
across raters. 


4.3.3 The role of self-assessment 


Trainees as well as practicing doctors are very poor self assessors**. Broad- 
ly speaking, self-assessment is a judgment one makes about oneself. Different 
authors have put forth different categories of self-assessment. Eva et al." have 
proposed to distinguish broad self-assessment (e.g. do I have a good sense 
of humour? Am I good enough in managing congestive heart failure?) from 
self-monitoring which is a more specific moment-to-moment evaluation of how 
one is doing during an activity. They suggest that broad self-assessment is main- 
ly relevant to continuing professional development which often relies on doctors 
to assess their learning needs and select appropriate activities, whereas self-mon- 
itoring is mainly relevant to autonomous practice (e.g. knowing when to look 
something up or refer a patient to someone else). Dory et al.** have proposed a 
four-category classification which distinguishes further between each of the two 
levels proposed by Eva et al., with the most general level referring to sweeping 
evaluations such as self-concept and self-esteem, and the most specific referring 
solely to metacognitive monitoring, i.e. monitoring of one’s mental processes. 
Self-assessment, particularly at the most general levels, has been found to be 
inaccurate. This is partly due to issues related to aggregating information from 
memories of several relevant episodes, but also to judgment biases such as the 
“above-average effect’, i.e. most people believe that they are above average in a 
wide variety of domains, which of course cannot be the case. 
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The practical implication is that self-assessment can never stand on its own 
and should always be triangulated with other information. Self-assessment 
should not on the other hand be discarded altogether. As a previously cited study 
has indicated, self-assessment influences reflection on feedback and coaches 
should pay deliberate attention to self-assessment when facilitating this process 
of reflection”. 


4.3.4 Formative and summative functions are typically combined 


Within a classical psychometric framework a clear distinction is made between 
formative and summative assessment procedures since both the design and the 
psychometric requirements of the test depend on the purpose of the assessment. 

Within a more “edumetric” framework that underpins assessment at the 
“does” level, an integration of formative and summative functions is advocated. 
Without formative value, the summative function would be ineffective, leading 
to trivialization or alienation of the assessment and introduce the risk of a neg- 
ative effect on the learning process. Also, “no single method can do it all”. Every 
assessment method and every single point assessment has its own limitations. 

Van der Vleuten & Schuwirth” therefore strongly advocate a “programmatic 
approach” of assessment in which assessment and learning are salient. Assess- 
ment in general and performance assessment in particular should be embed- 
ded in the curriculum (for trainees) or in daily practice (for licensed doctors). 
Such programmes of assessment cannot be improvised and should be planned, 
prepared, implemented, evaluated and improved. An assessment programme 
can be conceptualized as a purposeful collection of assessment moments, i.e. 
assessment data points. From the sum of those judgments a global picture will 
emerge that is more than the sum of the individual measurements. The pro- 
gramme should be aligned to the curriculum objectives (for trainees) and/or 
the job description (for practicing doctors). It should both foster learning and 
allow sound decisions to be reached. Within such programmes a meaningful 
aggregation across the available multiple assessment sources is advocated, that 
is triangulated with complementary judgments by external assessors ultimately 
leading to a defensible and robust summative pass/fail decision. 
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4.3.5 Qualitative, narrative information carries a lot of weight 


The assessment literature has been dominated over the last two decades by the 
psychometric paradigm and has focused on the development of reliable and val- 
id (normative) measurement instruments that produce scores based on quanti- 
fiable concept and quantitative methods. Within the complementary edumetric 
framework the main focus of assessment is shifted to producing meaningful 
feedback for learners. Successful feedback is conditional on social interaction. 
Feedback that consists of nothing more than quantitative information or a score 
is not very meaningful. Narrative and qualitative information should comple- 
ment and enrich the feedback. Effective formative assessment is predicated on 
qualitatively rich information. As a practical consequence rating forms should 
provide the possibility for assessors to provide complementary narrative feed- 
back. 


4.3.6 Summative decisions can be rigorous by using non- 
psychometric qualitative research procedures 


Within the proposed edumetric framework rigor can be defined in a similar 
way as in qualitative research. The concept of “trustworthiness” has been put 
forth”, and the conventional notion of internal validity is replaced by credibility, 
external validity by transferability, reliability by dependability and objectivity by 
conformability. 

Table 4 presents some examples of assessment strategies that mirror these 
trustworthiness strategies and criteria as they have been proposed by Van der 
Vleuten et al”. 


4.4 Miscellaneous issues 
4.4.1 Noise and signal in performance assessment 


A fundamental issue, far beyond any psychometric reasoning, is the question of 
the “nature” of medical competence and performance in family medicine. Is the 
GP who provides good quality care for the 90 percent of daily “trivial” problems 
performing better than the GP who demonstrates particular acumen in dealing 
with 10 per cent of particularly challenging conditions/clinical problems s/he 
will be confronted with? This philosophical question is at the heart of the matter 
in the debate on how performance should be measured, judged and evaluated. 
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Strategy to establish 
trustworthiness 


Credibility 


Transferability 


Dependability 
Confirmability 


Criteria 


Prolonged engagement 


Triangulation 


Peer examination 
Member checking 
Structural coherence 


Time sampling 


Thick description 
Stepwise replication 


Audit 


Potential Assessment Strategy (sample) 


Training of examiners 


Tailored volume of expert judgment based on 
certainty of information 


Benchmarking examiners 
Incorporate learner view 
Scrutiny of committee inconsistencies 


Judgment based on broad sample of data 
points 


Justify decisions 
Use multiple assessors who have credibility 


Give learners the possibility to appeal to the 


assessment decision 


Table 4: Potential strategies related to qualitative research methodologies for build- 
ing rigor in assessment decision 


We advocate that within a performance assessment programme both the core 
competencies as well as the more specific/advanced competencies should be 
looked at, reflecting the complexity of real practice. 


4.4.2 What is “unacceptable” performance? 


From a psychometric point of view medical competence has often been concep- 
tualized as a “continuous variable” and measured by more or less sophisticated 
instruments that produce a reproducible and valid score - an approach that can 
be challenged and debated. Is unacceptable or extremely poor performance a 
“normative” problem that has to be identified by a “cut-off” score (i.e. a mini- 
mum standard)? Or should “poor-unacceptable performance” of a licensed ex- 
perienced doctor be assessed using a more defensible qualitative approach, i.e. 
conceptualised as a categorical or nominal variable? Such an approach is based 
on the assumption that “unacceptable performance” is an extra-ordinary per- 
formance that cannot be captured within the logic of a “continuum” Therefore 
the term “poor performance” might have to be substituted by another term: for 
instance “aberrant” performance. We think the instruments aimed at identifying 
(or screening for) “aberrant” performance in a “rigorous” way, should be differ- 
ent from traditional instruments that have been put forth so far in the literature 
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and that such judgment should rely on holistic judgments by external assessors, 
using multi-source feedback, chart review, critical incident analysis and a per- 
sonal interview. 


4.5 Conclusion 


From the psychometric perspective workplace-based assessment offers new 
challenges. Traditional psychometric approaches tend to focus exclusively on 
quantitative properties of assessment outcomes. Within the proposed comple- 
mentary edumetric framework, inspiration is found in methodologies from 
qualitative research and the focus shifts to the gathering of rich contextual infor- 
mation and the role of assessment as a production tool of meaningful feedback 
to the learner. Broad sampling of meaningful and rich contextual information 
across educational or clinical contexts and across assessors remains essential in 
order to produce aggregated information from which - depending on the stakes 
of the performance assessment - a “robust” final judgment and decision can be 
made. 

Finally, performance assessment should be embedded in the curriculum (for 
trainees) or in daily practice (for licensed doctors). Such programmes of assess- 
ment cannot be improvised and should be planned, prepared, implemented, 
evaluated and improved. The EURACT Performance Agenda (EUPA) may help 
to shape and sharpen this process. 
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5 Primary care management 


Jean-Marie Degryse, Ruth Kalda, Roar Maagaard, 
Phil Phylaktou, Howard Tandeter, Peter Vajer, 
Yvonne van Leeuwen, Natalia Zarbailov 


Primary care management includes the ability: 

e To manage primary contact with patients, dealing with unselect- 
ed problems; 

e To cover the full range of health conditions; 

e To co-ordinate care with other professionals in primary care and 
with other specialists; 

* To master effective and appropriate care provision and health ser- 
vice utilisation; 

* To make available to the patient the appropriate services within 
the health care system; 

*  Toactas advocate for the patient. 


5.1 Introduction 


In everyday practice primary care management involves taking the history and 
systematic assessment of the patient, making a proper diagnosis, responding 
appropriately to the patient needs, cooperating with other team members and 
health care specialists in order to offer proper services for the patient as well as 
preventing unnecessary services and treatment. This includes the system of re- 
cord-keeping and information management and ways of organizing care within 
a practice or PHC team. 
Primary Care Management should reflect the following traits: 


e Knowledge about epidemiology of problems presented by patients in pri- 
mary care; 

e Understanding of the natural history of common conditions; 

e Skills for working with colleagues and in team; 

e Understanding of the structure, roles and responsibilities of the he primary 
health care team as being ever larger and more complex; 

e Understanding of the primary care organization and informational medical 
technologies; 
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e Ability to provide effective problem management with drug and non-drug 
approaches; 

e Effective communication skills; 

e Value the primary care approach; 

e Understanding of the probabilities and evolution of conditions in primary 
care which comes principally through experience therefore is often not fully 
developed until after formal training. 


5.2 Assessment methods 


Assessment of primary care management skills requires a variety of methods: 

Case based discussion (CBD), structured oral interview used in general 
practice, across a range of competency areas; the starting point is the written 
record of cases selected by the GP. 

Consultation observation tools (COT), tools to assess consultation skills, can 
be used to assess video recorded consultations or during direct observation in 
general practice settings. 

Multi-source feedback (MSF), assessment of clinical ability and professional 
behaviour, rated by five clinical colleagues, two occasions or rated by five clini- 
cal and five non-clinical colleagues on two occasions; needs skill of assessor in 
giving feedback. 

Naturally occurring evidence (NOE), from direct observation, “tagged” 
against appropriate competency headings, other practice-based activities, clini- 
cal supervisor's reports (CSR). 

Patient satisfaction questionnaire (PSQ), measures consultation and rela- 
tional empathy, can differentiate between doctors; needs skill of assessor in giv- 
ing feedback. 

Performance audit (PA) refers to an examination of a programme, function, 
operation or the management systems and procedures to assess whether the 
entity is achieving economy, efficiency and effectiveness in the employment of 
available resources. The examination is objective and systematic, generally using 
structured and professionally adopted methodologies. 

Review of patient records (RPR), review of the collection of documents that 
provides an account of each episode in which a patient visited or sought treat- 
ment and received care or a referral for care from a health care facility. 
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Simulated patient (SP), or standardized patient (SP) (also known as a patient 
instructor) in health care is an individual who is trained to act as a real patient 
in order to simulate a set of symptoms or problems. 


5.3 Documentation tools 


A variety of tools can be used: Patient satisfaction questionnaires, consultation 
assessment tools, portfolios etc. One possible example of assessment of chronic 
patient management is given below: 


Quality level: 
from 1 (is not done at all) to 5 (done appropriately) 


Characteristic 
Assessment of Patient’s Needs 1 2 3 4 5 
Goal Setting/Action Planning 
Problem-Solving 

Emotional Health Assessment 
Patient Involvement 

Patient Social Support 

Patient Self-management Education 
Continuity of Care 


Coordination of Referrals 


PPP PPP RP PRP RB 
NNN NY OU O 
Q OQ O WwW € CQ 
A BR A A A HR HR HK A 
gd oO I I RE O 


Case Documentation 


5.4 Abilities 


Ability 1: To manage primary contact with patients, dealing with 
unselected problems. 


Case vignette. 

Chris Butler, a man of 58 years of age, visits his GP as the last patient of a row of 
20, just before 6 pm. The patients of that afternoon had a variety of problems, e. g. 
onychomycosis, postpartum depression, diabetes mellitus type 2 and a tumour in 
the left breast. Ten however, had complaints that could easily be related to the flu, 
of which there is an epidemic outbreak at that moment. Chris Butler complains of 
“not feeling well” and aching of his thoracic muscles. He asks for “something against 
the flu”, because he does not want to take time off from work. The GP, however, 
asks him to answer some questions, first. The reason to do so, at this late hour, is 
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that this patient rarely visits the practice, has a family history of cardiovascular 
disease (several heart-attacks before the age of 60) and looks “off”. The history and 
physical examination increases the GPs gut-feeling that there is something serious 
going on: the patient reports that the muscular aching is exercise-related, his fore- 
head is moist, he has a tachycardia (100/min) and his blood pressure is 175/110 
mmHg. The wife is phoned by the practice nurse and the ambulance is ordered. The 
GP phones the cardiologist. 20 minutes later Chris Butler leaves the practice in the 
ambulance with the diagnosis: suspect of myocardial infarction. 


Features of this ability Assessment method Who could assess? 


Does the GP show his/her ability to include « Simulated patients * Staff 


facts form earlier contacts and family history? | , video observation e Trainer 


Does the GP interpret undefined complaints * Trainers opinion * Observers 
well, in spite of: low likelihood or misleading 


A : A « Test evaluator 
circumstances (flu epidemic)? 


Does the GP take time to carefully inter- 
rogation and examination despite the time 
pressure? 


Does the GP know morbidity figures related 
to patient characteristics? 


Table 5: To manage primary contact with patients, dealing with unselected problems 


Ability 2: To cover the full range of health conditions. 


The doctor has to bring a patient list for the previous (full) day - that is a list 
showing all the patients seen in the clinic by this doctor on the previous day. It 
should include: 

e Sex 

. Age; 

e Reason for encounter; 

e Medical records for the patients on the list. 


This list should be brought for discussion with the trainer/supervisor. 
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Features of this ability Assessment method Who could assess? 


A list demonstrating a variety of * Review of patient records * Trainer 
patients regarding sex, age and |. Discussion with supervisor | + Peer 
reasons for encounter trainer 


* Educational authority 
The doctor demonstrates in the 
medical records and in the dis- 
cussion that he is working with 
the problems presented to him. 


Table 6: To cover the full range of health conditions 


Ability 3: To co-ordinate care with other professionals in primary 
care and with other specialists. 


Case vignette. 

Monica Gustavsson, a 63 years old female patient with diabetes mellitus type 2, 
shows up with a chronic ulcer on the left lower limb. The ulcer has been treated 
according to the doctors instructions. Monica attends for a check-up after three 
weeks; the nurse makes the observation that she sees no improvement, telling the 
patient: "If things continue this way you are going to have serious problems, losing 
your leg because of the doctor" The nurse goes to doctor saying it is his fault that 
Monica Gustavssons wound is not improving. How should this situation be han- 
dled? 


Features of this ability Assessment method Who could assess? 
Minimum time spent in contact with * Patient satisfaction * Patient 

other professionals * Multisource feed- * Other medical profession 
Notes in the medical file of the opin- back (computerized |. Trainer 

ions of the others, or using integrated possibility) 


system of database, access to hospital | e Performance audit 
patients records 


Patient satisfaction in observing team- 
work 


Table 7: To co-ordinate care with other professionals in primary care and with other 
specialists 
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Ability 4: To master effective and appropriate care provision and 
health service utilization. 


Case vignette. 

The GP is called for neonatal visit to the 9 days old baby Caroline. She knows the 
family well. The mother is a 39 years old lady and during the first period of the 
pregnancy she supported flu. At the visit the parents look very sad and discouraged. 
According to the medical conclusion given from maternity hospital Caroline has 
serious health problems. 


Features of this ability Assessment method Who could assess? 

Does the GP think to involve « Discussion with GP * Trainer 

another care provider? * Direct or video observation * Peer 

Does the GP assess patient * Review of patient records * Observer 

needs? * Simulated patient * Other health care providers 
Does the GP has knowledge « Patient satisfaction ques- e Patient 

about health care structure and tionnaire 


he goes to contact the right 
persons, to use all available 
recourses? 


Does the GP have good com- 
munication with other health 
care providers? 


Table 8: To master effective and appropriate care provision and health service 
utilization 
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Ability 5: To make available to the patient the appropriate 
services within the health care system. 


Case vignette. 

Maria Parini, a 56 years old female patient visits her GP because of chronic low 
back pain lasting more than four months. She was examined thoroughly by her 
GP and has no serious disease needing neurosurgery. She has sleeping problems, is 
tired because of pain, depressed and seems disturbed. Maria Parini works as sales- 
clerk in a little shop, and has difficulties being on her feet all day. She uses NSAIDs, 
which have little or no effect. 


Features of this ability Assessment method Who could assess? 


Does the GP show good communication skills « Simulated patients ¢ Simulated patient 
covering all possible affects of the pain to the 

patient everyday life: physiologic, psychological . . à : 
and psychosocial aspects? * Discussion with trainer | * Peer 


* Video observation * Trainer 


Does the GP take enough time for detection 
possible psychosocial, environmental or social 
barriers to recovery (job dissatisfaction, depres- 
sion or anxiety disorder etc.)? 


Does the GP involve to the treatment spine- 
care education and active exercise pro- 
grammes using physiotherapists, back-schools, 
cognitive-behavioural support or other available 
resources? 


Does the GP use tricyclic antidepressants as 
approved treatment of chronic back pain? 


Table 9: To make available to the patient the appropriate services within the health 
care system 
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Ability 6: To act as advocate for the patient. 


Case vignette. 
Dusan Smilic is a 75 years old male patient, who has coxarthrosis (OA). His GP 
prescribed him NSAIDs, which are covered by his health insurance. NSAIDs were 
effective in relieving the pain and improving the functioning, but the patient also 
consulted an orthopaedist, who suggested using cox-2-inhibitors which are actually 
not covered by the health insurance. Dusan Smilic now asks his GP what he should 
do. You as a GP are sure that there is no necessity to change the treatment which is 
effective and cheap into a treatment with similar effectiveness but being 10 times 
more expensive. 

How should the GP demonstrate now the role of advocate of the patient? 

He should contact the specialist and ask why the specialist decided to change 
the treatment, explain his own personal opinion regarding this issue, also the 
patients problems relating the more expensive treatment. 

Then he should suggest the patient not to change the treatment and explain 
why there is no need. 


Features of this ability Assessment method Who could assess? 


Does the GP demonstrate active | + Simulated patients * Trainer 
advocacy against the third party - Video observation 


* Simulated patient 
(in this case orthopaedist)? ? 


* Direct observation * Observer 
* Peer 


Table 10: To act as advocate for the patient 
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6 Person-centred care 


Okay Basak, Elena Frolova, Sandra Gintere, Filipe Gomes, 
Eva Jurgova, Monica Lindh, Markku Timonen, Adam Windak 


Person-centred care includes the ability: 

e To adopt a person-centred approach in dealing with patients and 
problems in the context of patient’s circumstances; 

e To apply the general practice consultation to bring about an ef- 
fective doctor-patient relationship, with respect for the patient's 
autonomy; 

e To communicate, set priorities and act in partnership; 

e To provide longitudinal continuity of care as determined by the 
needs of the patient, referring to continuing and co-ordinated 
care management. 


6.1 Introduction 


General Practice/Family Medicine positions itself as a “person-centred medi- 
cine” with commitment to the person rather than to a particular body of knowl- 
edge, seeking to understand the context of the illness, and attaching importance 
also to the subjective aspects of medicine”. 

Person centred approach is more a way of thinking than just a way of acting. 
It means seeing the patient always as a particular person in a particular context 
and it includes a total health perspective of a patient, not only the disease ele- 
ments that can be recognised in the problems and complaints???5, 

In this complex process, evolving over time, the doctor-patient relationship 
is often called "sustained partnership” - referring to a mutual and full engage- 
ment over longer periods and to the patient's participation and responsiveness. 

The patient centred clinical method is at the base of general practice/family 
medicine. Its main elements are: 


e Understanding the person as a whole - an integrated human person in his 
or her context; 

e Exploring always illness and disease in relation to the person and the con- 
text, taking into account the patient's ideas, feelings and expectations; 
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e Finding common ground that can lead to mutual plans, keeping the patient 
in the centre of all decisions and respecting his or her autonomy; 

e Acting in a partnership with asymmetric but defined roles and responsibil- 
ities, always enhancing the doctor-patient relationship; 

e Incorporating prevention and health promotion in balance with all ele- 
ments of disease management; 

e Being realistic about time and resources and act in a concrete and solu- 
tion-focused way”. 


Person centred care places a great importance on the continuity of the re- 
lational process. Continuity covers five domains: over time continuity (the 
chronological domain), geographical continuity (at the one location of the clinic 
and the practice), interdisciplinary continuity (the primary care team), interper- 
sonal continuity and informational continuity (guaranteeing the availability of 
medical information). 


6.2 Assessment methods 


Assessing performance elements that every general practitioner/family physi- 
cian (GP/FP) should master implies the utilisation of different methods appli- 
cable in the GP/FM setting. Some - but not all - ofthe appropriate methods in 
person centeredness are listed below: 


e Patient view (direct questions, questionnaires); 
e Simulated patient; 

e Direct observation/sitting-in; 

e  Video/video recorded consultation; 
e Consultation maps; 

e Consultation rating scales; 

e Group discussion; 

e Written case report; 

e Chronological case progression; 

e Essay; 

e Reflective diary/portfolio; 

e Chart audit. 
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6.3 Documentation tools 


Several tools have been developed in order to facilitate the practical assessment 
of consultations oriented towards person/patient centeredness. 

Consultation maps and consultation rating scales” are widely used in the 
practice setting. Score sheets are used in a teaching/learning context. Examples 
of the above are shown at the end of this chapter. 


6.4 Abilities 


Ability 1: To adopt a person-centred approach in dealing with 
patients and problems in the context of patient's circumstances. 


Case vignette, step 1. 

José Silva, male, 48 years old, comes to the GP office because he is worried about 
the results of a PSA test done within a preventive programme at his workplace. He 
asks for advice on what should be the next step. 


Medo ars gemon Features of this ability Assessment method ie canik 
strate this ability? assess? 
Scientific knowledge * Knowledge of concepts/ * Essay * Trainers 
foundations of person * Group discussion * Peers 
centred care 
* Relevant literature 
Reference frame * Understanding of cultural * Chart audit * Trainers 
and gender differences * Reflective portfolio * Peers 
« Mastering family assess- - Written case report 
ment tools (genograms, 
family plots, eco-mapping) 
Mastering patient illness + Exploring both disease * Chart audit * Trainers 
and disease concepts and illness - Patient view (direct e Peers 
questions, question- e Patients 
naires) 
Self awareness of doctor + Counter-transference * Video * Trainers 
* Self-awareness of emo- * Direct observation/ * Peers 
tional responses sitting-in 


Group discussion 


Reflective diary/portfolio 


Table 11: To adopt a person-centred approach in dealing with patients and problems 
in the context of patient's circumstances 
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Comment - Adopting a person-centred approach allowed the Doctor to deal 
with the problem as experienced and presented by the Patient. 

Doctors "agenda" - mostly concentrate in predictive values of tests and ad- 
equacy of preventive interventions - was encompassed with Patient's worries. 


Ability 2: To apply the general practice consultation to bring 
about an effective doctor-patient relationship, with respect 
for the patients autonomy. 


Case vignette, step 2. 
José Silva is a mathematics teacher in secondary school. He divorced three years 
ago and remarried one year ago with a 28 years old colleague, now three months 
pregnant. Until now he has been a rather healthy man, with no chronic diseases 
and no medication. He never smoked, and is a moderate drinker. He takes regular 
exercise three times per week, i. e. tennis and cycling. There are no cancer or early 
cardiovascular diseases in his family history. 

Physical examination including DRE did not reveal any abnormalities except 
for slight overweight (BMI 27). Brought in laboratory tests results: blood count, 
blood glucose, lipidogram, liver function, CRP are normal; PSA 6.0. 
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How do GPs demon- A Th Who could 
strate this ability? Features of this ability Assessment method FECE) 


Patient centred con- e Nature, history, aetiology + Consultation maps * Trainers 
sultation model and effects of problems * Consultation rating scales - Peers 


adequately defined : : 
. . * Video recorded consultation 
Patient's ideas, concerns 


and expectations ade- * Chart audit 
quately explored * Direct observation/sitting-in 


Patient's context and 
circumstances took into 
account 


Common ground pur- 
sued and found in rela- 
tion to problems, goals 
and roles 


Mutual decision 
achieved 


At risk factors consid- 
ered 


Relationship with patient 
enhanced 


Appropriate use of time 
and resources 


Report findings in a Shared understanding of + Patient view (direct ques- * Patients 
adapted and under- problems achieved tions, questionnaires) 


standable way * Direct observation 
* Videos 


* Trainers 


Take decisions with Appropriate action cho- e Video recorded consultation + Trainers 
respect for the auton- sen for each problem e Chart audit e Peers 
omy of the patient 


Patient involved in man- 


* Group discussion * Team 
agement 


* Patient view (direct ques- * Patients 
tions, questionnaires) 


Table 12: To apply the general practice consultation to bring about an effective doc- 
tor-patient relationship, with respect for the patient's autonomy 


Comment - The knowledge of both sensitivity and specificity of PSA measure- 
ments, and of the epidemiology and natural course of prostatic cancer were 
essential to this case. The use of a patient-centred clinical model for the con- 
sultation clarified the situation and the context of the patient, leading to the ap- 
propriate mutual decision. Preventive issues were balanced and discussed. Lab- 
oratory findings were understood and the patient was involved in management. 
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Ability 3: To communicate, set priorities and act in partnership. 


Case vignette, step 3. 

José Silva is afraid of prostate cancer and possible death; going deeply into the in- 
terview he displays problems of erectile dysfunction since he saw the laboratory 
results. 


How do GPs demon- A En Who could 
strate this ability? Features of this ability Assessment method — 


Effective exchange of + Listening to the patient's «+ Simulated patient * Trainers 


information point of view e Video recorded consultation + Peers 


* Giving information * Direct observation/sitting-in «+ Patients 


* Checking understanding — , patient view (direct ques- 


tions, questionnaires) 


Acceptance of pa- * Considering patient's * Patient view (direct ques- * Patient 
tient's points of view feelings, needs and tions, questionnaires) - Trainers 
expectations oxi ; 

B Simulated patient -Peers 


e Video recorded consultation 
« Direct observation/sitting-in 


Mutual involvement in + Involving patient on * Written case report * Trainers 


health care plan decision making + Simulated patient e Peers 


* Finding common ground — , video recorded consultation + Patient 
on problems and goals . . A MH 
* Direct observation/sitting-in 


Table 13: To communicate, set priorities and act in partnership 


Comment - The effective exchange of information (implying doctors own emo- 
tional responses as well) and the doctor/patient relationship enhancement made 
possible to bring up the underlying feelings of the Patient, therefore displaying 
the main problem. 


Ability 4: To provide longitudinal continuity of care as 
determined by the needs of the patient, referring to 
continuing and co-ordinated care management. 


Case vignette, step 4. 

José Silva is reassured about his PSA results, after second determination and ul- 
trasound check. No longer fearing cancer, he sees his erectile malfunction problem 
disappear. Three months later his young wife comes with him to the GP’ office. 
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She is six months pregnant now. José Silva is now fully informed about preventive 
activities and risk groups - namely of quaternary prevention! 

Quaternary prevention: Action or set of actions aiming to identify patients at 
risk of overmedicalization, protect them from invasive medical procedures and 
propose ethically acceptable interventions’. 


Bes Nn- Features of this ability Assessment method a E 
Personal continuity * Communication e Video recorded consultation + Trainers 
* Doctor/patient relation- * Direct observation/sitting-in * Patients 
ship « Patient view (direct ques- 
tions, questionnaires) 
* Reflective portfolio 
Continuity over time * Documentation/files/ * Written case report * Trainers 
charts * Chronological case pro- 
* Follow-up plan gression 
* Chart review 
* Video recorded consultation 
* Direct observation/sitting-in 
Medical information * History taking * Chart audit * Trainers 
* Record keeping * Group discussion * Peers 
* Retrieving information e Video recorded consultation + Team 
Interdisciplinary * Collaboration with * Chart audit * Trainers 
team-members * Group discussion * Peers 
* Use of other resources « Written case report e Team 
* Chronological case pro- 
gression 


Table 14: To provide longitudinal continuity of care as determined by the needs of 
the patient, referring to continuing and co-ordinated care management 


Comment - The fact that the patient could maintain his relationship with his 


doctor over time was essential to the success. Interdisciplinary collaboration was 
considered but not needed. 
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6.5 Examples of tools used to facilitate 


the assessment of consultations 


6.5.1 Consultation map 


Useful tool to monitor the evolution of a patient centred consultation. 

Each time the doctor or the patient speaks, a mark is placed against the appro- 
priate heading. These marks may then be joined together so that the sequence of 
events in the consultation is made clear. 


54 


Nature and history of problems 


Aetiology of problems 


Patient’s ideas 


Patient’s concerns 


Patient’s expectations 


Effects of problems 


Continuing problems 


At risk factors 


Action taken 


Sharing understanding 


Involving in management 


6.5.2 Consultation rating scale 


Evaluates consultations, rating them according to a scale, by placing marks in 
such a position along each line to show how much the person in charge of the 


evaluation agrees with each statement. 


Nature and history of problems 
adequately defined 


Nature and history of problems 
defined inadequately 


Aetiology of problems 
adequately defined 


Aetiology of problems defined 
inadequately 


Patient’s ideas explored 
adequately and appropriately 


Patient's concerns explored 
adequately and appropriately 


Ideas explored inadequately or 
inappropriately 


Concerns explored inadequately 
or inappropriately 


Patient's expectations explored 
adequately and appropriately 


Expectations explored 
inadequately or inappropriately 


Effects of problems explored 
adequately and appropriately 


Effects of problems explored 
inadequately or inappropriately 


Continuing problems considered 


Continuing problems not 
considered 


At risk factors considered 


At risk factors not considered 


Appropriate action chosen for 
each problem 


Inappropriate action chosen 


Appropriate shared 
understanding of problems 
achieved 


Shared understanding not 
achieved or inappropriate 


Patient involved in management 
adequately and appropriately 


Appropriate use of time and 
resources in consultation 


Involvement in management 
inadequate or inappropriate 


Inappropriate use of time and 
resources in consultation 


Use of time and resources 
in long-term management 
appropriate 


Inappropriate use of time 
and resources in long-term 
management 


Helpful relationship with patient 
established or maintained 


Unhelpful or deteriorating 
relationship with patient 
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6.5.3 Score sheets 


Although usually used in learning/teaching environment, score sheets as the one 
presented below can easily be adapted to practice assessment. 


SCORE SHEET - CONSULTATION 
(Swedish specialist exam in Family Medicine, 2006) 


Sitting in Video Examiner’s notes 


Patient: age, sex, description 


Starting time — ending time = time used (minutes) - = min 


A Comprehensive (Global) 
Themes 

Patients agenda 

Dr’s agenda 

Use of time 


PO);RONP 


Opening - rapport 


2. Patient’s presentation of symptom/ 
problem 


3. Patient's own idea/assessment of 
problem, expectations, concerns 


4. Identify problem, focus. 


5. Physical examination 


6. Use of medical service/ 
investigations 


7. Relevant preventive aspects 


8. Summarize — assess-test hypothesis 


9. Mutual assessment and plan. 
Closure. 


Assess according to A and/or according to B. Notes are used to compile the examiner’s report. 


A comparable tool has been published in The Netherlands as LACONTO 
instrument. 
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7 Specific problem solving skills 


Mette Brekke, Bernard Gay, Givi Javashvili, Janko Kersnik, 
Razvan Miftode, Maryna Oliynik, Mladenka Vrcic-Keglevic 


Specific problem solving skills include the ability: 

e To relate specific decision making processes to the prevalence 
and incidence of illness in the community; 

e To selectively gather and interpret information from history-tak- 
ing, physical examination, and investigations and apply it to an 
appropriate management plan in collaboration with the patient; 

e To adopt appropriate working principles, e.g. incremental inves- 
tigation, using time as a tool and to tolerate uncertainty; 

e To intervene urgently when necessary; 

e To manage conditions which may present early and in an undif- 
ferentiated way; 

« To make effective and efficient use of diagnostic and therapeutic 
interventions. 


7.1 Introduction 


Specific problem solving skills in family practice relate to the context in which 
the problems are encountered, the nature and natural history of the problems 
themselves, the personal characteristics of the patients presenting with these 
problems, personal characteristics of the doctors who manage them, and the 
resources we have at our disposal to manage these problems. Problem solving in 
general practice is highly context-specific. 

The task of problem solving when faced with early undifferentiated illness 
requires the GP to use a problem-based approach rather than a disease based 
approach. Several good primary care textbooks are organized along these lines. 

Use of time, incremental investigation and coping with uncertainty are part 
of the attitudinal change that may be necessary for those learning general prac- 
tice. There is a growing body of literature on these topics to support trainers who 
encourage learners to reflect on these unique aspects of problem solving. 

Both shared management of problems with the patient and the conflict over 
the fair use of limited resources raise ethical issues connected with problem 
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solving. The trainer of general practice can focus attention to ethics when ap- 
propriate in specific cases or may present case simulations to provoke discussion 
of these problems and coping strategies. 

A large part of specialty training is devoted to problem solving skills. Over 
a period of several months or years the trainee will develop a unique personal 
clinical style under the guidance of a tutor. Problem solving is best taught in the 
clinical setting with actual patients followed over time to assess outcomes. 


7.2 Assessment methods 


Assessment of the performance of problem solving skills in primary care re- 
quires a variety of methods. Direct (sit-in with real patients; video-taped con- 
sultation) or indirect (e.g. chart review) observation of practice using checklists 
and global ratings can be used. The performance of decision making can be as- 
sessed by performance review. Patient interviews or questionnaires can be used 
to assess patient satisfaction with the doctor's attempts to involve them in their 
care. 


7.3 Documentation tools 


A list for the observer, not related to specific clinical conditions, but used for 
overall assessment of problem solving skills: 
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The assessor should observe and evalu- Never Most of 
ate if the GP being assessed is able to the time 


1. Have in mind the epidemiological 0 1 2 3 4 
situation in society, so that he first 
considers the most probable reasons 
for any complaint 


2. Selectively collect appropriate infor- 0 1 2 3 4 
mation 

3. Use properly the specific information 0 1 2 3 4 
gathered 

4. Carry out appropriate investigations 0 1 2 3 

5. Involve patients in decision making, 0 1 2 3 
and respect patients' preferences 

6. Take into account psychological, 0 1 2 3 4 
social and family factors 

7. Empower the patient, make him feel 0 1 2 3 4 
well and valued 

8. Make use of previous knowledge of 0 1 2 3 4 
the patient 

9. Be open-minded about that any com- 0 1 2 3 4 


plaint in some cases may not be what 
he first assumed 


10. Use time as a diagnostic tool 0 1 2 3 
11. Act appropriately in case of an emer- 0 1 2 3 
gency 
12. Maintain proper organisation in the 0 1 2 3 4 


practice and proper use of time 


A mix of patients should be observed in order to get a final score on the score 
sheet above. 

These points could be an observational sheet for a peer observer. Few single 
consultations will involve all these points, but during a day one surely will be 
able to evaluate all the points. 


7.4 Abilities 


Case vignette. 
Kurt Schmidt, a previously healthy man aged 20 years in a Western European 
country contacts his GP for a cough lasting three weeks. 
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Case vignette. 
Susanne Brioche, a 51 years old woman contacts her GP because of epigastric pain 
and nausea lasting for two weeks. She asks for a referral to a computer tomogram. 


Case vignette. 
Sara Campos, a previously healthy 33 years old woman expecting her first child 
shows up for pregnancy control in her 34th week. Her blood pressure has increased 
to 160/105 mmHg, she has protein ++ in her urine and is feeling generally unwell. 


Case vignette. 
Natasa Pontosic, a 47 years old woman contacts her GP because of tiredness, pain 


in her head and stomach and numbness in her legs. 


Ability 


Ability 1: 
To relate 
specific 
decision 
making 
processes 
to the prev- 
alence and 
incidence 
of illness in 
the com- 
munity 


60 


How do GPs 
demonstrate 
this ability? 
(case vignette) 


Kurt Schmidt, a 
previously healthy 
man aged 20 
years in a West- 
ern European 
country contacts 
his GP for a 
cough lasting 
three weeks. 


Features of this ability 


* History taking of symp- 
toms related to the most 
probable illnesses 


History taking to rule out 
red flag symptoms 


Physical examination 
for the signs of the most 
probable illnesses 


Ordering of just the ap- 
propriate test 


To make decisions 
according to the most 
frequent reasons for the 
contact: appropriate man- 
agement plan (including 
watchful waiting, patient 
involvement, patient 
consent, appropriate risk 
taking) 


Assessment 
method 


* Direct or indirect 
observation of 
the first contact 
of the patient 
with symptoms: 
sitting in, video- 
tape 


Chart review 


Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


For use not only 
as a normative 
assessment, but 
as a formative 
assessment, 
with subsequent 
feed-back to 

the GP 


Who could 
assess? 


* Peers 


* Supervi- 
sor 


* Tutor 
* Trainer 


Ability 


Ability 2: 
To se- 
lectively 
gather and 
interpret 
information 
from histo- 
ry-taking, 
physical 
examina- 
tion, and 
investi- 
gations 
and apply 
it to an 
appropriate 
manage- 
ment plan 
in collabo- 
ration with 
the patient 


Ability 3: 
To adopt 
appropriate 
working 
principles, 
e.g. in- 
cremental 
investiga- 
tion, using 
time asa 
tool and to 
tolerate un- 
certainty 


How do GPs 
demonstrate 
this ability? 
(case vignette) 


Kurt Schmidt, a 
previously healthy 
man aged 20 
years in a West- 
ern European 
country contacts 
his GP fora 
cough lasting 
three weeks. 


Susanne Brioche, 
a 51 years old 
woman contacts 
her GP because 
of epigastric 

pain and nausea 
lasting for two 
weeks. She asks 
for a referral to a 
computer tomo- 
gram. 


Features of this ability 


* Selectively collecting 
appropriate information 


* Not using time on issues 
not related to the actual 
situation 


* Carrying out a focused — 
not a general — physical 
examination 


Explaining a management 
plan and making sure the 
patient understands and 
feels comfortable with it 


Showing good communi- 
cation skills 


Use an appropriate 
amount of time 


* Elaborating a manage- 
ment plan including 
simple treatment and 
watchful waiting. 


Involving the patient in the 
diagnostic reasoning and 
the management plan 


Exploring the patient's 
fear of having a serious 
condition 


Making the patient feel 
secure that she will be 
taken care of 


Not referring the patient 
for investigations with no 
proper indication 


Making reappointment for 
follow-up 


Referring for appropriate 
investigations when a 
condition shows an unex- 
pected course 


Assessment 
method 


* Direct or indirect 
observation of 
the first contact 
of the patient 
with any condi- 
tion: sitting in, 
videotape 

* Chart review 


* Review of re- 
ferrals 


* Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


* Direct or indirect 
observation of 
first contact of 
the patient with 
any condition, 
especially those 
involving diffuse 
and unspecific 
symptoms 

* Chart review 


* Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


Who could 
assess? 


Peers 


Supervi- 
Sor 


Tutor 
Trainer 


Peers 


Supervi- 
Sor 


Tutor 
Trainer 
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Ability 


Ability 4: 
To in- 
tervene 
urgently 
when nec- 
essary 


Ability 5: 
To manage 
conditions 
which may 
present 
early and 
in an undif- 
ferentiated 
way 


How do GPs 
demonstrate 
this ability? 
(case vignette) 


Sara Campos, 

a previously 
healthy 33 years 
old woman ex- 
pecting her first 
child shows up 
for pregnancy 
control in her 
34" week. Her 
blood pressure 
has increased to 
160/105 mmHg, 
she has protein 
++ in her urine 
and is feeling 
generally unwell. 


Natasa Pontosic, 
a 47 years old 
woman contacts 
her GP because 
of tiredness, pain 
in her head and 
stomach and 
numbness in her 
legs. 


Features of this ability 


« Focused history taking of 
symptoms with focus on 
alarm signs and symp- 
toms — red flags 


* Physical examination with 
focus on alarm signs — 
red flags 


* Immediate treatment or 
referral to appropriate 
level of care 


« Reassuring but realistic 
communication with the 
patient about diagnostic 
thinking and management 
plan 


* Appropriate use of time 


* Adapting attitudes char- 
acteristic of a generalist 
orientation, including 
curiosity, diligence and 
caring 


* Adapting a stepwise 
procedures in medical 
decision-making, using 
time as a diagnostic and 
therapeutic tool 


* Understanding the in- 
evitability of uncertainty 
and apply strategies to 
tolerate uncertainty. This 
will require: 

* Making follow-up ap- 
pointment 
* Making the patient feel 


comfortable with the 
follow-up plan 


* Involving the patient in 
your diagnostic thinking 


* Instructing the patient 
to make earlier contact 
in case of worsening 


Assessment 
method 


* Direct or indirect 
observation of 
first contact of 
the patient with a 
probable serious 
condition requir- 
ing urgent action 


* Chart review 


* Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


* Direct or indirect 
observation of 
first or subse- 
quent contact of 
the patient with a 
condition related 
to ill defined 
symptoms, which 
might resolve or 
which may also 
turn out to be a 
serious condition 


* Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


Who could 
assess? 


* Peers 


* Supervi- 
sor 


* Tutor 
* Trainer 


* Peers 


* Supervi- 
sor 


* Tutor 
* Trainer 
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Ability 


Ability 6: 
To make 
effective 
and effi- 
cient use 
of diag- 
nostic and 
therapeutic 
interven- 
tions 


How do GPs 
demonstrate 
this ability? 
(case vignette) 


Kurt Schmidt, a 
previously healthy 
man aged 20 
years in a West- 
ern European 
country contacts 
his GP fora 
cough lasting 
three weeks. 


Features of this ability 


« Behaving according 
to an understanding 
of cost-benefit of tests 
and treatments, and the 
number needed to treat 
or harm for specific treat- 
ments: 


Not ordering expensive 
test when cheaper ones 
are available 


Not prescribing drugs that 
are unnecessary costly for 
the patient or for society 


Assessment 
method 


Direct or indirect 
observation of 
consultation with 
a patient with 
any acute or 
chronic condition 


Chart review 


Review of re- 
ferrals 


Review of lab 
ordering 


Review of drug 
prescriptions 


Review of a 
logbook with 
defined number 
of the manage- 
ment of defined 
cases 


Table 15: Specific problem solving skills, abilities 1-6 


Who could 
assess? 


Peers 


Supervi- 
sor 


Tutor 
Trainer 
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8 Comprehensive approach 


Owen Clarke, Natasa Pilipovic Broceta, Smiljka Radic, 
Mario R. Sammut, Stefan Wilm 


Comprehensive approach includes the ability: 

« To manage simultaneously multiple complaints and pathologies, 
both acute and chronic health problems in the individual; 

« To promote health and well being by applying health promotion 
and disease prevention strategies appropriately; 

e To manage and co-ordinate health promotion, prevention, cure, 
care and palliation and rehabilitation. 


8.1 Introduction 


One of the important requirements for family doctors is to be able to address 
multiple complaints and problems in the patients they care for, and at the same 
time support the individual strengths and resources to cope with these prob- 
lems. When patients feel they need medical assistance, they become ill as a per- 
son and often cannot differentiate between different possible diseases they may 
have. The challenge to address all the multiple health issues in an individual is 
an important one. It requires an important skill of interpreting the issues and 
prioritising them in consultation with the patient. 

The family doctor should aim at an approach to the patient where the main 
focus would be in promoting their health and global well being, which is of- 
ten in sharp contrast with the specialist approach in treating as many medical 
problems as possible. Adequate handling of risk factors by promoting self-care 
and empowering patients is an important task of the general practitioner. The 
aim of the family doctor is to minimise the impact of patient's symptoms on his 
well-being by taking into account his personality, family, daily life and physical 
and social surroundings. Adoption of an evidence-based approach should pro- 
vide the patient with currently best-documented treatment, and should provide 
the doctor with currently best-documented evidence for diagnosis and treat- 
ment. 

Coordination of care also means that the family physician is adequately 
skilled not only in managing disease and prevention, but also in caring for the 


65 


patient and providing palliative care in the end phases of the patients’ lives and 
providing rehabilitation. The physician must be able to coordinate patient care 
that is provided by other health care professionals. 


8.2 Assessment methods 


Assessing GP performance requires knowledge of what happens in real everyday 
practice. Thus, appropriate methods for assessing the comprehensive approach 
should include: 


e Observation by peers: sitting in, video and audio recordings, one-way mir- 
ror; 

e Audit of medical records; 

e Feedback from patients/relatives. 


Such assessment may be: 
e Cross-sectional (observation of a few patients); 
e Longitudinal (several encounters in medical records). 


8.3 Documentation tools 


The methodology makes use of a data collection form, consisting of columns 
with these headings: patient/consultation number, chronic problem list, prob- 
lems presented, health promotion, prevention, cure, palliation, rehabilitation, 
coordination in the team and referral. 


Patient/ 
consultation 
number 
Chronic 
problem list 
Problems 
presented 
Health 
promotion 
Prevention 
Cure 
Palliation 
Rehabilitation 
Co-ordination 
& referral 


The form is used to collect data from a series of different patients on the 
same day in observed consultations, or to summarise information from a series 
of consultations with the same patient over a period of time. The rows therefore 
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can represent either consultations with different patients or different consulta- 
tions with the same patient. The same form can gather either real-time (live) 
data or historical data from records. 

While the form can be used as a form of self-assessment by the GP who 
completes it, performance assessment requires that a selection of the data (for 
example, three different cases or consultations) is discussed with the GP in detail 
by the assessor/s. Such assessors have to be experienced GPs, because a non-GP 
would not have the competence to assess the comprehensive approach. 

The following vignette in three parts illustrates how performance assessment 
of the three abilities of the comprehensive approach takes place in practice. 


8.4 Abilities 


Ability 1: To manage simultaneously acute and 
chronic health problems in the individual. 


Case vignette. 

Amanda Miller is a 73 year old obese woman with polyarthropathy and atrial 
fibrillation, for which she takes multiple medications. She lives with and looks after 
her husband who has dementia. Amanda Miller comes to see her GP stating that 
she has “influenza”. 


Features of this ability Assessment method Who could assess? 
« Recognises and tackles acute problem « Observation * Peer 
* Keeps in mind chronic problems simultane- 
ously 
* Avoids drug interactions * Observation * Peer 
* Recognises home situation * Medical records 
* Use of medical records * Medical records * Peer 
* Addresses patient concerns, differentiating * Feedback * Patient 


between wants and needs e Peer 


* Agrees with patient on management plan 


Table 16: To manage simultaneously acute and chronic health problems in the 
individual 
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Ability 2: To promote health and well being by applying health 
promotion and disease prevention strategies appropriately. 


Case vignette (continued). 
One month later, Amanda Miller comes to see her GP again, this time regarding 
her joint pains. 


Features of this ability Assessment method Who could assess? 


Identifies opportunity * Observation * Peer 
for health promotion 
intervention (e.g. using 
the 5 A's tool) 


Makes use of struc- 
tured medical record as 
reminder of preventive 
activities 


* Medical records 


Makes use of team- 
work 


Uses communication * Observation * Peer 
skills 


Is aware of any per- 
sonal prejudices which 
might influence nega- 
tively preventive advice 


* Makes recommen- * Observation * Peer 
dations which are e Medical records e Patient 
feasible in the context 
of patient's lifestyle and * Feedback 
home situation 

* Acts as suitable role * Observation * Peer 
model e Patient feedback * Patient 


Table 17: To promote health and well being by applying health promotion and dis- 
ease prevention strategies appropriately 
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Ability 3: To manage and co-ordinate health promotion, 
prevention, cure, care and palliation and rehabilitation. 


Case vignette (continued). 
Another two months later, Amanda Miller falls at home and suffers a fractured hip. 
After surgery, she is sent home from hospital. 


Features of this ability Assessment method Who could assess? 


« Provides appropriate * Observation * Peer 


medical management e Medical records 


Involves members of 
practice team, e.g. 
through case confer- 
ence 


Performs/arranges 
home visit (consider 
relevant team mem- 
bers) — needs assess- 
ment, e.g. cause of 


falls 
* Involves relatives * Medical records * Peer 
* Feedback * Patient/relatives 
* Facilitates communi- * Medical records * Peer 
cation (two-way) with e Case discussion 
hospital 


Involves external com- 
munity resources (e.g. 
home help, etc.) 


Table 18: To manage and co-ordinate health promotion, prevention, cure, care and 
palliation and rehabilitation 
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9 Community orientation 


Dolores Forés, Roger Price, Llukan Rrumbullaku, 
Alma Eir Svavarsdottir, Paula Vainiomäki, Egle Zebiene 


Community orientation includes the ability: 

e To reconcile the health needs of individual patients and the 
health needs ofthe community in which they live in balance with 
available resources. 


9.1 Introduction 


Family doctors have a responsibility for the community in which they work, 
which extends beyond the consultation with an individual patient. The work of 
the family doctor is determined by the makeup of the community and therefore 
the doctor must understand the potentials and limitations of the community. 
The family doctor is in a position where he/she can see these issues. In all soci- 
eties health care systems are being rationed. Doctors are being involved in the 
rationing decisions, and have an ethical and moral responsibility to influence 
health policy in the community. 


9.2 Assessment methods 


e Records review to review achievement against target; 

e Records review to assess procedures to address needs of the absent resi- 
dents; 

e Case report, case description; 

e Observation of doctor and team in action. 

Who could assess? 


e Self assessment, internal audit; 
« External audit - review of process through peer appraisal. 


9.3 Documentation tools 


e . Check lists; 
e Observation lists; 
e Consultation maps. 
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9.4 Ability 


Ability: To reconcile the health needs of individual patients and 
the health needs of the community in which they live in balance 
with available resources. 


Case vignette. 

The primary health care team arrives at the nursing home of 42 residents to per- 
form flu vaccinations. Three residents are in outpatient clinic and want the immu- 
nization; others get the vaccination, except two residents who do not like vaccina- 
tion as they are afraid of needles. The GP demonstrates her community orientation 
by assessing the needs of the small nursing home community as compared to the 
needs of these two residents, including their health needs, but also their personal 
preferences. The decision of the doctor should be aimed at persuading those two 
persons to agree with the vaccination, as it affects not only them, but also increases 
the risk of disease and also complications for other residents, who may be severely 
ill. This aim should be balanced with respecting the two patients’ individual auton- 
omy. The plan of the team is to return the next day for vaccination of the missing 
three residents. 


How do GPs demon- 


strate this ability? Features of this ability ne ANCES 


(case vignette) method assess? 
All residents in a nursing + Organization * Record * Self as- 
home should get annual «+ Administration review sessment 
flu vaccination unless * Collaboration with * System * Internal 
contraindicated by EbM nursing home health review audit 
care providers e Case report «+ External 
* Record keeping * Case de- audit 
* Goal: to immunize scription 
100% of eligible nurs- + Observation 
ing home inhabitants of doctor in 
action 


Table 19: To reconcile the health needs of individual patients and the health needs 
of the community in which they live in balance with available resources 
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10 Holistic approach 


Francesco Carelli, Georgi Ivanov, Bernhard Rindlisbacher, 
George Spatharakis, Wolfgang Spiegel 


Holistic modelling includes the ability: 
e To use a bio-psycho-social model taking into account cultural 
and existential dimensions. 


10.1 Introduction 


Holistic care is understood as an approach to patient care in which all the pa- 
tient’s physical, mental and social factors are always taken into account, rather 
than just the diagnosed disease with or without psychosocial factors that influ- 
ence the disease or the result. 

Procedure for holistic care: 


1. To obtain information about the whole of the patient (all aspects of the per- 
son, his well-being, social and professional integration, his hopes, aspira- 
tions and fears and all the biological, psychological and social aspects of his 
existence) and his interrelations to the rest of the of the world/the system. 
These pieces of information are like pieces in a puzzle without borders and 
never complete. 

2. Synthesize all the collected information (the pieces of the puzzle) and arrive 
at a comprehension of how all aspects interact with the patient’s well-being, 
social and professional integration, aspirations and fears. 

3. To produce and offer an individualized management plan to the patient and 
family concerning all three levels, comprising not only therapeutic recom- 
mendations, but also counselling, integrated care and interventions with 
alternative input/choices. 


Medicine is an intrinsic part of the wider culture. It is based on a set of shared 
beliefs and values, as with any cultural practice. The definition of holistic ap- 
proach that is widely accepted for medical care, and will be used in this docu- 
ment, implies 


caring for the whole person in the context of the person’s values, his family be- 
liefs, the family system, and the culture and the socio-ecological situation in the 
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larger community, and considering a range of therapies based on the evidence of 
their benefits and cost. 


Holism, as Pietroni states, involves a 


willingness to use a wide range of interventions... an emphasis on a more partic- 
ipatory relationship between doctor and patient; and an awareness of the impact 
of the “health” of the practitioner on the patient. 


This individual focus makes it relate closely to family medicine. The holis- 
tic view acknowledges objective scientific explanations of physiology, but also 
admits that people have inner experiences that are subjective, mystical and (for 
some) religious, which may affect their health and health beliefs. 

The recognition that all illnesses have both mental and physical components 
and that there is a dynamic relationship between components of systems (gen- 
eral systems theory) led to the development of the bio-psycho-social model of 
modern medicine. The position of the biopsycho-social model was spelled out 
most clearly by George L. Engel who argued that for psychiatry to generate a ful- 
ly scientific and inclusive account of mental disorder, bio-reductionist accounts 
should be superseded by ones which adhere to the insights of the general sys- 
tems theory, developed by Ludwig von Bertalanffy and Paul Weiss. The bio-psy- 
cho-social model was proclaimed as a paradigm shift because it apparently dis- 
solved the mind-body split. 

Understanding the illness (not disease) as a process, which gives equal im- 
portance to biological, psychological and social determinants for pathogenesis, 
diagnosis and therapy, forms the holistic approach with its consequent imple- 
mentation to practical measures. 

Using a bio-psycho-social model as the basis for cure and care implies an 
acceptance that many factors influence our understanding of what it is to be 
human. Family doctors accept a diversity of factors to be of relevance. Examples 
of factors may be: 


e The natural disposition, including elements of gender, genetic constitution 
and typology; 

e The micro-social environment such as the family and the macro-social en- 
vironment, including the local community and the wider community with 
all its cultural and socio-ecological elements; 

e The health beliefs and life experiences that make a person the entity that he/ 
she is now; 
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e Health-maintaining resources in a person, like the understanding of events, 
the acceptance of meaning, the autonomy that leads to the conviction that 
life is manageable; 

e Personal experiences including past illnesses, medical and social contacts. 


As the list of factors grows, it is also important to stress that a basic awareness 
and understanding of one’s own limitations as a doctor are crucial. Keeping in 
mind the fundamental autonomy of the patient, there is a limited opportunity 
for the family doctor to intervene occasionally and rather “tangentially”, with an 
interesting but very scarce knowledge about the person’s history, feelings and 
priorities. At the same time, the integration of influencing factors is crucial and 
constitutes the added value. This refers to system approach, where the whole is 
considered more than the sum of the parts’. 


10.2 Assessment methods 


The information that gives an idea of how a GP really works as it concerns the 
holistic approach, could be received by two kinds of methods: direct and indirect. 


10.2.1 Direct 


e Sitting in with GP 
Criteria used: 
e Facilitation of patients and giving time and interest to express personal 
problems; 
e Attitude versus time/ability to find time to show the attitude; 
e Portfolio of the patient (“pointed” image of the patient like in puzzles); 
e Michael Balints"* approach use: What is the presenting symptom, abil- 
ity/failure to find what is behind it? 
e  Videotaped consultation analysis; 
e Simulated consultation; 
e Interviews with 
e Patients; 
e Members of the family; 
e Doctors; 
e Caregivers (formal and informal); 
e Other medical staff in the practice; 
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Other non-medical staff in the practice; 
Others (patients’ friends, other involved persons, etc.); 


e Interviews with peers or specialized professionals. 


10.2.2 Indirect 


Medical records consultation for search of information concerning the three 
fields/levels. These, besides clinical assessment, should include: 


1. Psychological assessment (symptoms, problems, presence of anxiety, de- 
pression, etc.) 
2. Social assessment 


e 


e 


Information about family composition and living conditions; 
Satisfaction with personal/family life; 
Knowledge about family ties and relationships; 
Family tree knowledge; 

Individual and family financial situation; 
Profession and information on professional life; 
Satisfaction at work; 

Level of education; 

Religion and beliefs; 

Spiritual life/needs; 

Social class and life. 


A holistic method to assess the ability of the doctor to use an holistic ap- 
proach might be to let the doctor role-play the patient with his problems and 


have him draw a picture of the patient. 


10.3 Documentation tools 


A broad range of documentation tools may be used to document the results of 
the above assessment methods, including check lists, scoring tools, interview 
protocols, observation lists and consultation maps. 
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10.4 Ability 


Ability: To use a bio-psycho-social model taking into account 
cultural and existential dimensions. 


Case vignette. 
The family GP responds to a house call to see and assess Victor Lazlos, a known 
terminal cancer patient of 78 years of age who feels very tired and exhausted these 
last days, and has lost his abilities to conduct the activities of daily living. The 
patient lives in-doors for some months now and has no social contacts, while the 
family assumes full responsibility of the care. History taking, clinical examination 
and biology reveal no major disturbances, the disease passing through a phase of 
relative stability. Victor Lazlos has a depressive expression, and the discussion as 
well as the use of the GDs-30 scale confirms the diagnosis of a severe depression. 
During the visit the patient’s daughter has to leave to go shopping and asks 
the grandson to take care of his grandfather while she will be absent. This is done 
in front of the patient who feels hurt and useless, a burden to the family. Further 
discussion reveals that Victor Lazlos is a strong religious believer. 
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How do GPs demon- 
strate this ability? 


The personalized and 
holistic plan that is 
produced in agree- 
ment with the patient 
and his family: 

1) Regular doctor 
visits and biologi- 
cal control 

2) Local home aid 
group visits three 
times a week for 
adequate follow-up 
and help to basic 
daily activities 
(bathing, etc.) 

3) Treatment for 
depression 

4) Counselling of the 
family to role-play 
as if they asked 
the help of the 
grandfather to look 
upon the grandson 
while the daughter 
is absent, and 
not the opposite 
(this will value the 
patient and make 
him feel useful and 
will give a meaning 
to his life) 

5) Organize the 
house visit of a 
priest — monk, who 
is known and well 
respected by the 
patient 

6) Organize escapes 
from the house en- 
vironment as soon 
the condition of the 
patient permits it 

7) Mobilize volunteers 
to visit and social- 
ize with the patient 
and to escort him 
when later going 
out of the house 


Features of this ability 


Some global (indirect) 
criteria such as: 

* Patient's satisfaction 
* Family's satisfaction 
* Doctor's satisfaction 


Information in medical 

charts on all three levels: 

for example checklists 

for: 

i. Spiritual needs 

ii. Financial situation 

iii. Sexual life 

iv. Satisfaction at work 

v. Satisfaction with 
personal/family life 


Decision making and de- 
cision sharing approach 
(creation with patient 
and family of a manage- 
ment plan) — readiness 
of the patients to open 
up and trust the doctor 
to his questioning and 
suggestions 


Looking after all the 
members of the family 
and take in account their 
suggestions (if possible 
by social conditions) 


Family calls the GP for 
matters not only related 
to biological health, but 
also for psycho-social 
problems — the patient 
shares completely his/ 
her thoughts with the 
doctor and capacity to 
evoke changes 


Rate of referral of the 
patients and families to 
the local social services 
(this is close to the 
notions of chapters 4 
and 5 of the Educational 
Agenda) 


Openness, acceptance, 
empathy 


Assessment method 


* Sitting in with GP 

* Videotaped consultation analysis 

* Simulated consultation 

* Interviews with: patients, family, 
other doctors, caregivers (formal 
and informal), other medical staff 
in the practice, other non-med- 
ical staff in the practice, others 
(e.g. patients’ friends) 

* Interviews with peers or special- 
ized professionals 


Medical records’ consultation for 
search of information 


Sitting in with GP 

Videotaped consultation analysis 
Simulated consultation 

nterviews with: patients, family, 
other doctors, caregivers (formal 
and informal), other medical staff 
in the practice, other non-med- 
ical staff in the practice, others 
(e.g. patients’ friends) 

* Interviews with peers or special- 
ized professionals 


* Sitting in with GP 

* Videotaped consultation analysis 

* Interviews with: family, other 
doctors, caregivers (formal and 
informal), other medical staff in 
the practice, other non-medical 
staff in the practice 


* Medical records’ consultation for 
search of information 

* Interviews with: patients, family, 
other doctors, caregivers (formal 
and informal), other medical staff 
in the practice, other non-medi- 
cal staff in the practice 


* Medical records’ consultation for 
search of information 

* Interviews with specialized 
professionals 


* Role playing 
* Sitting in with GP 
* Videotaped consultation analysis 


Who could assess? 


Internal 

* Self-assessment 
* Work colleagues 
External 

Special audit from: 
* Peers 

* Quality assurance 

groups: 

1) Associations of GPs 
2) Delegates of insur- 
ance companies 

3) State institutes 


Table 20: To use a bio-psycho-social model taking into account cultural and existen- 
tial dimensions 
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11 Glossary of terms and key assessment methods 


In the present EURACT Performance Agenda, a lot of educational terms are 
used. In the addendum readers can find a short glossary on some of these ed- 
ucational terms, predominantly taken from the EURACT Educational Agenda. 
Description of types of assessment is partially based on Newble and Cannon™. 
Key assessment methods presented in the Agenda are described in the last para- 
graph (terminology on assessment tools and methods). Arrows (>) refer to oth- 
er terms in the glossary. 


11.1 More general terminology 


Basic medical education, BME 

The part of the medical curriculum that relates to all medical students, to 
give a sound basis for further vocational or specialty training. The European 
Union claims a minimum of six years of BME. 


Bachelor degree 

Following the Bologna Declaration 1999 to create a “European Higher Educa- 
tion Area” by 2010, the first basic part in each >curriculum should preferable 
have 3 years and 180 study points. 


Continuing medical education, CME 
Any and all ways by which a graduated physician continues to learn and change 
in practice in a lifelong learning scheme. 


Continuing professional development, CPD 

A process of planned and individually tailored learning in practice with a focus 
on the quality of care. CPD includes the identification of learning needs, con- 
struction of a learning agenda, drawing a concrete learning plan, and controlling 
this in an educational portfolio format. As it relates to lifelong learning, it can 
become a lifelong Personal Development Plan’. 


Convergence 
Voluntary adoption of suitable policies for the achievement of a common goal. 


Credit 
The “currency” used to measure students’ >workload in terms of the notional 
learning time required to achieve specified >learning outcomes. 
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Credit framework 

A system that facilitates the measurement and comparison of learning out- 
comes achieved in the context of different qualifications, programmes of study 
and learning environments. 


Credit level 

An indicator of the relative demand of learning and of >learner autonomy. It 
can be based on the year of study and/or on course content (e. g. Basic, Interme- 
diate, Advanced, and Specialized). 


Credit type 

An indicator of the status of course units in the programme of study. It can be 
described as Core (major course unit), Related (unit providing instruments/sup- 
port) and Minor (optional course unit). 


Curriculum 

In formal education, a curriculum is the planned interaction of learners with 
instructional content, materials, resources, and processes for evaluating the at- 
tainment of educational objectives. The general practice curriculum in —vo- 
cational training defines the learning outcomes for the specialty of general 
practice and delivers a full description of the ^ knowledge, skills, attitudes and 
behaviours required of a GP in managing patients and their problems. 


Doctorate or Doctoral degree 

A high level qualification which is internationally recognized as qualifying 
someone for research or academic work. It will include a substantial amount of 
original research work which is presented in a thesis. It is generally referred to as 
the degree awarded after completion of third cycle studies. 


ECTS (European Credit Transfer System) 

A system for increasing the transparency of educational systems and facilitating 
the mobility of students across Europe through credit transfer. It is based on 
the general assumption that the global >workload of an academic year of study 
is equal to 60 credits. The 60 credits are then allocated to course units to describe 
the proportion of the learners’ workload required to achieve the related >learn- 
ing outcomes. Credit transfer is guaranteed by explicit agreements between the 
home institution, the host institution and the mobile >learner. 
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First degree qualification 

First higher education qualification taken by the >learner. It is awarded after 
successful completion of first cycle studies which, according to the Bologna Dec- 
laration should normally last a minimum of three years or 180 — ECTS credits. 


(Initial) Master degree 

Following the Bologna Declaration 1999 to create a "European Higher Educa- 
tion Area" by 2010, after obtaining first a bachelor degree, a second part in the 
curriculum leads to a master degree. It should have a minimum of 2 years 
and 120 — ECTS-study points. For medical master studies, a clinical period of 
3 to 4 years is accepted in many European countries. Obtaining a master degree 
normally includes some form of master thesis. 


Master after Master or postinitial Master degree 

Following the Bologna Declaration 1999 to create a "European Higher Educa- 
tion Area" by 2010, after obtaining an —initial master, programmes can give 
entrance to a postinitial master programme. In this logic, specialty training 
should be considered a postinitial master programme. 


Specialty training 
vocational training 


Tuning 

Developing agreement and harmony by combining single sound into a common 
"tune" or pattern of sounds. It is used in the "tuning project" to achieve a form of 
harmonization by finding points of convergence and common understanding. 


Vocational training, VT, syn. specialty training, ST 

The part of the medical curriculum that comes after the common basic 
medical education programme for all medical students, and focuses on the ac- 
quisition of the >competences, required for the specialty discipline and related 
tasks in healthcare. 


11.2 Terminology on educational content 


Attachment 
A period of longer attachment in a practice setting, also called “preceptorship” 
or “clerkship”. 
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Competence 

The capability to successfully perform discrete observational tasks in a defined 
—assessment environment, in isolation from actual work. In the Miller termi- 
nology, it includes the level of “knowing” (basic facts), “knowing how” (able to 
apply knowledge) and “showing how” (able to apply >knowledge) but it ex- 
cludes the “doing” level, the >performance in practice. 


Elective course 
A course to be chosen from a predetermined list. 


Knowledge 

Facts, information, and skills acquired through experience or education by per- 
ceiving, discovering, or learning; the theoretical or practical understanding of 
a subject. It can be implicit (as with practical skill or expertise) or explicit (as 
with the theoretical understanding of a subject); it can be more or less formal or 
systematic. Knowledge acquisition involves complex cognitive processes: per- 
ception, communication, association and reasoning. 


Learners 

Refers to students in basic medical education (BME) as well as >vocational 
trainees (VT) or practicing doctors in >continuing medical education (CME) 
or their >continuing professional development (CPD) and to all those who 
take part in the training programmes. 


Learning outcomes 

Statements on what a >learner is expected to know, understand and/or be able 
to demonstrate after completion of a process of learning. Learning outcomes are 
distinct from the aims of learning, in that they are concerned with the achieve- 
ments of the learner rather than the overall intentions of the >teacher. Learn- 
ing outcomes must be accompanied by appropriate assessment criteria which 
can be used to judge that the expected learning outcomes have been achieved. 
Learning outcomes together with assessment criteria specify the minimum re- 
quirements for the award of —credit, while marking is based on attainment 
above or below the minimum requirements for the award of credit. 


Mark 
Any numerical or qualitative scale used to describe the results of assessment 
in an individual >course unit or module. 
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Mentor 
tutor 


Objectivistic learning 

Traditional education model, based on ^ knowledge transfer from teacher 
to learner. It is highly teacher centred. Content is structured in handbooks, 
teaching is mainly focused on lecturing by experienced teachers, and behaviour 
copied from experienced role models. 


Performance 

The level of actual performance in clinical care and communication with patients 
in daily practice. It relates in the Miller terminology to the “doing” level. It is con- 
sidered highly dependent on existing healthcare conditions and requirements, 
financial and structural opportunities, practice opportunities and support. 


Problem based learning 

Educational model that takes the problem of the patient and the doctor as the 
starting point for the learning curriculum. It is highly learner centred, op- 
timizes the use of pre-existing 2 knowledge, and stimulates self learning and 
search strategies. 


Social constructivistic learning 

Educational model that puts the learning process of the student as the central 
point. Learning is seen as a process, highly dependent on pre-existing >knowl- 
edge and on learning context. ^ Teachers are mainly architects of the stimu- 
lating learning environment for the learners. Individual variety in learning 
strategies are stimulated. 


Teacher 
Refers to all professionals involved in an educational event as experts. 


Tutor, syn. mentor, facilitator 
A professional involved in the educational process as leader of the process, to 
guide and reflect to the benefit of the >learner(s). 


Workload 
All learning activities required for the achievement of the >learning outcomes 
(i.e. lectures, practical work, information retrieval, private study, etc.). 
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11.3 Terminology on learning methods 


Clinical work/clinical practice under supervision 

Teaching during working in clinical environment, in general practice/family 
practice/primary care setting. It can be organised with or without >supervi- 
sion. 


Courses 
Structured programme of educational content, often presented in an oral for- 
mat, supported by course material. 


Discussion 

Discussion session on a specific topic or case presentation. Can be organised as 
a one to one session with >tutor or >supervisor, a peer group session, a small 
group session like a focus group or a Balint group, or it can be a large/temporary 
group session at seminars, lectures or ^ workshops. 


Interactive (IT based) learning 

Combination of modular reflection packages, linking case studies, focused 
reflection, discussion forums, library search and/or reflection in one educa- 
tional process. 


Lecture 
Provision of teaching content by presentation and explanation (possibly includ- 
ing a demonstration) by a lecturer. 


Literature search 

Learning to perform a medical database search in EbM, including defining a 
clinical question, looking for medical evidence, critical reflection on evidence 
and implementation in practice. 


Observation 

Learning through reflective observation by a —tutor/— supervisor in different 
educational settings: sit-in with real patients or simulated patients; learning 
through video-taped consultation of real patient or simulated patient (observa- 
tion by oneself, tutor/supervisor, peers, etc). 


Reading/studying 
Reading books, protocols, EbM information, novels, narratives, internet etc. 
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Reflection 
On self, e.g. by using a diary, videos, or in a participative reflection group. 


Role playing 

Using the act of playing a role as a patient, as an accompanying person, as a 
doctor, as a nurse etc to derived educational insight in feelings, intentions and 
actions. 


Project work 
Working out a personal project or as part of a group in a defined format: >audit 
project, research project, field work project etc. 


Seminar 
A period of instruction based on written or oral contributions by the >learners, 
usually in small groups. 


Skills training 

Learning procedural skills in adapted specific settings like: doing procedures 
e.g. in a skills lab, learning consultation skills by e.g. >role playing, learning 
(medical) database searching, learning leadership skills by running an educa- 
tional or targeted meeting etc. 


Study visit 
Educational visit to a practice, to clinical premises, to social-welfare institutions, 
to health authorities etc. 


Supervision/supervisor 
Supervision involves regular, ongoing structured meetings/sessions with and 
—feedback from personal >tutor/supervisor on various topics. 


Writing of patient studies, case studies 
Educational activity, with a given task to provide a written description and/or 
reflection document, to get >feedback from a >tutor/>supervisor. 


Workshop 
A supervised session where learners work on individual tasks and receive as- 
sistance and direction when needed. 
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11.4 Terminology on assessment tools and methods 


Appraisal 

A structured process of facilitated self >reflection!” and a means of aiding GP 
personal development. Key roles may include annual exploration of role expec- 
tations; a review of progress towards previously agreed objectives; a recognition 
of achievements; and an identification of personal development needs’. 


Assessment 

The total range of written, oral and practical tests, as well as projects and >port- 
folios, used to decide on the learner’s progress in the course unit or in a 
module. These measures may be mainly used by the >learner to assess his/her 
own progress (formative assessment) or by the teacher responsible to judge 
whether the course unit or module has been completed satisfactorily against the 
learning outcomes of the unit or module (summative assessment). 


Assessment criteria 
Descriptions of what the learner is expected to do, in order to demonstrate 
that a >learning outcome has been achieved. 


Audit 

A planned and documented activity performed by qualified personnel to deter- 
mine by investigation, examination, or evaluation of objective evidence, the ad- 
equacy and compliance with established procedures, or applicable documents, 
and the effectiveness of implementation. Audit in healthcare is a process used 
by health professionals to assess, evaluate and improve care of patients in a sys- 
tematic way, from simple data collection e.g. in medical records to the more 
complex sequence of steps entailed in completing the full-cycle (initial audit, 
change implemented, re-audit to demonstrate improvement). Audit measures 
current practice against defined (desired) standard and explicit criteria. It forms 
part of clinical governance, which aims to safeguard a high quality of clinical 
care for patients. The key component of clinical audit is that 9 performance is 
reviewed (or audited) to ensure that what should be done is being done, and if 
not it provides a framework to enable improvements to be made. 


Blueprint 
Bringing the relative importance of different clinical areas, covered in an —as- 
sessment procedure in accordance with the large variety of cases and problems 
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and their prevalence in real practice. In the broad range of GP problems, blue- 
printing is important, because of the problem of >case specificity. 


Case based discussion (CBD) 
Structured oral interview across a range of competency areas; the starting point 
is the written record of cases selected by the trainee’. 


Case specifity 

Research has shown that learning in medicine is very much case-specific. Mas- 
tering a limited set of cases and/or problems does not guarantee the mastering 
of other cases, areas and problems, especially in a large field like GP/FM. So 
—assessment based on the handling of one or a few cases only gives a very 
restricted information on the >competence of a candidate. Assessment with 
10 small tasks of five minutes selected with a good >blueprint generally gives 
much more valid information than one long case of 50 minutes. 


Checklist 

A list of competencies to be mastered at the end of a training period, format- 
ted as a clear defined list, checkable by the learner, by the teacher or by 
both, providing a constant overview of what is already mastered and what is still 
to be learned. 


Consultation observation tools (COT) 
Tools to assess consultation skills, can be used to assess video recorded consulta- 
tions or during direct observation in general practice settings'”. 


Essay method 

Written — reflection on specific questions, in the extended response kind (de- 
scribe what should be done for...) or the restricted response kind (given this 
statement, describe this specific issue). Problem is the time needed and the low 
reliability. 


Feedback 

e. g. by patients, relatives of patients, or staff. There is now convincing evidence 
that systematic feedback delivered by a credible source can change clinical 
performance, although there are many complexities that influence the effec- 
tiveness of feedback in practice"? (>multi-source feedback). 
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MCQ-MEQ method 

Multiple Choice Questionnaire: a format of objective measurement of the 
knowledge of the >learner. Later adapted to other formats: Modified Essay 
Questionnaire, the Extended Matching type, etc. MEQ includes clinical reason- 
ing, not only knowledge testing. 


Multi-source feedback 

A unique form of >workplace-based assessment in that it uses a collection 
of untrained raters, and the feedback based on the collated ratings is subse- 
quently fed back to the trainee or doctor by the supervisor or peer. Thus it has 
aspects of assessment of and for learning. 


Objective test method 

Includes a wide variety of test formats, in which the marking or the answer is ob- 
jective: true/false questions; Multiple Choice Questions; context-dependent 
questions, where a degree of analysis is needed to find the answers; extended 
matching questions, i.e. more complex combination of themes, scenarios, wide 
range of possible options, sometimes in relation to specific conditions. 


Observation method, direct 

Direct observation of >performance on technical or interpersonal skills 
in the real, simulated or examination setting: sit-in with real patient in clinical 
practice or >simulated patients (SOO, simulated office oral); video-taped con- 
sultation with real patients in clinical practice or simulated patients with stan- 
dardised assessment'’*. Observation by oneself, >tutor/>supervisor, peers, 
etc. >Valid method, but >reliability is low. It can be made more objective by 
the use of >checklists, rating forms, and training the examiners. 


Observation method, indirect 
Simulates direct observation e.g. by using patient records (chart audit), med- 
ical certificates, progress reports or by using patient-case discussion. 


On-site peer assessment 
—workplace-based assessment, > peer (group) assessment 


Oral method 
Traditionally the most used method, with a high face validity, but very time 
consuming and unreliable. Remedies are standardisation of the content (by clear 
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definition, by selection of a standard set, by using >standardised patients etc), 
or reducing examiners inconsistency (rating sheets, multiplying examiners with 
independent marking). 


OSCE method 

Objective Structured Clinical Examination: a format of objective evaluation, fo- 
cused on (complex) skills testing through lists of wanted and unwanted features 
in relation to the skill. 


Patient satisfaction questionnaire 
Provides patient +feedback on doctors’ empathy and relationship-building 
skills during consultations'^. 


Peer (group) assessment 

— Assessment is done by peers and not by —tutors/— supervisors. Different 
formats can be used. One specific is the 360 degree assessment format, where 
at least 10 colleagues, health personnel and staff contribute to the assessment. 


Portfolio, educational or reflective 

A portfolio is a summary of the major teaching activities and accomplishments, 
in relation with a Curriculum Vitae, including products and publications. It be- 
comes a reflective or educational portfolio by adding a reflective part, where the 
—learner reflects on the personal learning process!'*. It can be used in a printed 
or in an electronic version. 


Reliability 

Refers to the reproducibility of the scores on the assessment; high score reliabil- 
ity indicates that if the test were to be repeated over time, examinees would receive 
about the same scores on retesting as they received the first time. Unless assess- 
ment scores are reliable and reproducible (as in an experiment) it is nearly impos- 
sible to interpret the meaning of those scores - thus, validity evidence is lacking. 


Review 
e.g. of medical records (audit) 


Review of patient records (RPR) 

Review of the collection of documents that provides an account of each episode 
in which a patient visited or sought treatment and received care or a referral for 
care from a health care facility. 
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Self assessment method 

Evaluation method to help learner’s understanding of own ability and >perfor- 
mance. Criteria and standards are defined in a series of small group meetings 
by staff and >learners. Then learners use the criteria to judge their own perfor- 
mance. 


Short answer method 
Judging by asking specific short answers on given clinical vignettes: what is the 
diagnosis, list two typical symptoms, etc. 


Simulated patient 

Simulated patient (SP), or standardized patient (SP) (also known as a patient 
instructor) in health care is a healthy subject, or an actual patient who has been 
trained to portray accurately and consistently a particular patient case in order 
to simulate a set of symptoms or problems, and who is also trained to assess the 


— performance of students, trainees or doctors based on pre-defined criteria”. 


Structured written answer method 

After a given variable amount of patient data follows a series of options, between 
which the learner has to select the requested answer. Two types: the Patient 
Management Problem and the —^ Modified Essay Questionnaire on a broader 
field of possible options. 


Validity 

Refers to the evidence presented to support or refute the meaning or interpreta- 
tion assigned to >assessment results. All assessments require validity evidence 
and nearly all topics in assessment involve validity in some way. Validity is the 
sine qua non of assessment, as without evidence of validity, assessments in med- 
ical education have little or no intrinsic meaning. 


Video observation 
— observation method, direct 


Workplace-based assessment (WPBA) 

WPBA is the —assessment of competence and performance based on 
what a trainee or doctor actually does in the workplace. The main aim of WPBA 
is to aid learning and reflecting (assessment for learning) by providing trainees 
and doctors with constructive feedback and to support development. Train- 
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ees and doctors can use the same methodology to assess themselves (reflective 
practice). The assessments help the supervisors, >tutors and peers to chart 
a trainee’s progress during a placement or a doctor’s performance in daily prac- 
tice. One major advantage of workplace-based assessment is its ability to evalu- 
ate performance in context!!%119120121122,123,124,125,126 (_> on-site peer assessment). 
Saturation can be sought by applying a variety of assessment methods, by 360 
degree assessment (peer (group) assessment) and/or by longitudinal assess- 
ment (>multi-source feedback). 


91 


12 


13 


14 


References 


Allen J, Price E, Svab I, Windak A. Framework for continuing education- 
al development of trainers in general practice/family medicine in Europe. 
Cracow: EURACT, 2012. 

WHO. World Health Report. Primary health care: now more than ever. Ge- 
neva: WHO, 2008. 

Starfield B. Primary care: an increasingly important contributor to effec- 
tiveness, equity, and efficiency of health services. SESPAS report 2012. Gac 
Sanit 2012;26(Suppl1):20-6. 

Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

Starfield B. Primary care: Balancing health needs, services and technology. 
Oxford: Oxford University Press, 1998. 

Fahy N. Who is shaping the future of European health systems? BMJ 
2012;344:e1712. 

Allen J, Price E, Svab I, Windak A. Framework for continuing education- 
al development of trainers in general practice/family medicine in Europe. 
Cracow: EURACT, 2012. 

WONCA Europe. European definition of family medicine. 2002. 
Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

EURACT Statement on Undergraduate Teaching in Family Medicine 
in all European Universities, April 2014. http://www.euract.eu/others/ 
finish/20-others/299-euract-statement-on-undergraduate-teaching-in- 
family-medicine-in-all-european-universities-april-2014 

Wilm S. From medical school to retirement. Eur J Gen Pract 1996;2(2):77- 
81. 

Allen J, Price E, Svab I, Windak A. Framework for continuing education- 
al development of trainers in general practice/family medicine in Europe. 
Cracow: EURACT, 2012. 


93 


15 


16 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


27 


28 


94 


Brekke M, Carelli F, Zarbailov N, Javashvili G, Wilm S, Timonen M, Tan- 
deter H. Undergraduate medical education in general practice/family med- 
icine throughout Europe - a descriptive study. BMC Medical Education 
2013;13:157. 

Basak O, Yaphe J, Spiegel W, Wilm S, Carelli E, Metsemakers JF. Early clin- 
ical exposure in medical curricula across Europe: an overview. Eur J Gen 
Pract 2009;15(1):4-10. 

Tandeter H, Carelli FE, Timonen M, Javashvili G, Basak O, Wilm S, Zarbailov 
N, Spiegel W, Brekke M. A “minimal core curriculum" for family medicine 
in undergraduate medical education: A European Delphi survey among 
EURACT representatives. Eur J Gen Pract 2011;17:217-20. 

Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

Wilm S, ed. EURACT checklist for attachment program organizers. Euro- 
pean Academy of Teachers in General Practice Leuven: EURACT, 2005. 
http://www.euract.eu/resources/specialist-training 

Allen J, Price E, Svab I, Windak A. Framework for continuing education- 
al development of trainers in general practice/family medicine in Europe. 
Cracow: EURACT, 2012. 

http://www.euract.eu/resources/cme-cpd 

Miller GE. The assessment of clinical skills/competence/performance. Acad 
Med 1990;65(Suppl.):S63-7. 

Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDu- 
ca T, et al. The relationship between competence and performance: Implica- 
tions for assessing practice performance. Med Educ 2002;36:901-9. 
Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDu- 
ca T, et al. The relationship between competence and performance: Implica- 
tions for assessing practice performance. Med Educ 2002;36:901-9. 
Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDu- 
ca T, et al. The relationship between competence and performance: Implica- 
tions for assessing practice performance. Med Educ 2002;36:901-9. 
Rethans JJ, Norcini JJ, Baron-Maldonado M, Blackmore D, Jolly BC, LaDu- 
ca T, et al. The relationship between competence and performance: Implica- 
tions for assessing practice performance. Med Educ 2002;36:901-9. 
Campion P. Can you compare competence and performance? A reply to 
McKinstry et al. Family Practice 2004;21:81-2. 


29 


30 


31 


32 


33 


34 


35 


36 


37 


38 


39 


40 


41 


Rethans JJ, van Leeuwen Y, Drop R, van der Vleuten C, Sturmans F. Compe- 
tence and performance: Two different concepts in the assessment of quality 
of medical care. Family Practice 1990;7(3):168-74. 

Starfield B. Toward international primary care reform. Can Med Assoc J 
2009;180(11):1091-2. 

Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

Finucane PM, Bourgeois-Law GA, Ineson SL, Kaigas, TM. A comparison of 
performance assessment programs for medical practitioners in Canada, Aus- 
tralia, New Zealand, and the United Kingdom. Acad Med 2003;78:837-43. 
Schuwirth LWT, Southgate L, Page GG, Paget NS, Lescop JMJ, Lew SR, et al. 
When enough is enough: a conceptual basis for fair and defensible practice 
performance assessment. Med Educ 2002;36:925-30. 

Van der Vleuten CPM, Schuwirth LWT, Driessen EW, Dijkstra J, Tigelaar D, 
Baartman LKJ, et al. A model for programmatic assessment fit for purpose. 
Medical Teacher 2012;34:205-14. 

Parkerton PH, Smith DG, Belin TR, Feldbau GA. Physician performance 
assessment: nonequivalence of primary care measures. Medical Care 
2003;41(9):1034-47. 

Finucane PM, Bourgeois-Law GA, Ineson SL, Kaigas, TM. A comparison of 
performance assessment programs for medical practitioners in Canada, Aus- 
tralia, New Zealand, and the United Kingdom. Acad Med 2003;78:837-43. 
Finucane PM, Barron SR, Davies HA, Hadfield-Jones RS, Kaigas TM. To- 
wards an acceptance of performance assessment. Med Educ 2002;36:959-64. 
Schuwirth LWT, Southgate L, Page GG, Paget NS, Lescop JMJ, Lew SR, et al. 
When enough is enough: a conceptual basis for fair and defensible practice 
performance assessment. Med Educ 2002;36:925-30. 

Rethans JJ, Gorter S, Bokken L, Morrison L. Unannounced standardised 
patients in real practice: a systematic literature review. Medical Education 
2007:41:537-49. 

Overeem K, Faber MJ, Arah OA, Elwyn G, Lombarts KM, Wollersheim HC, 
et al. Doctor performance assessment in daily practise: does it help doctors 
or not? A systematic review. Med Educ 2007;41:1039-49. 

Govaerts M et al. Broadening perspectives on clinical performance assess- 
ment: Rethinking the nature of in-training assessment. Adv in Health Sci 
Educ 2006;12:239-60. 


95 


42 


43 


44 


45 


46 


47 


48 


49 


50 


51 


52 


53 


54 


55 


96 


Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and as- 
sessment of clinical skills of medical trainees: a systematic review. JAMA 
2009;302:1316-26. 

Pelgrim E et al. In-training assessment using direct observation of sin- 
gle-patient encounters: a literature review. Adv in Health Sci Educ 
2011;16:131-42. 

Miller A, Archer J. Impact of workplace based assessment on doctors’ edu- 
cation and performance: a systematic review. BMJ 2010;341:c5064. 

Van den Hombergh P, Grol R, van den Hoogen HJ, van den Bosch WJ. 
Practice visits as a tool in quality improvement: acceptance and feasibility. 
Qual Health Care 1999;8:167-71. 

WONCA Europe. European definition of family medicine. 2002. 
Heyrman, J, ed. EURACT Educational Agenda. Leuven: European Acade- 
my of Teachers in General Practice EURACT, 2005. 

EQuiP, EURACT. Continuing professional development in primary health 
care. Quality development integrated with continuing medical education. 
Policy Document of EQuiP and EURACT, 2002. 

Miller GE. The assessment of clinical skills/competence/performance. Acad 
Med 1990;65(Suppl.): 563-87. 

Norcini JJ. Current perspectives in assessment: the assessment of perfor- 
mance at work. Med Educ 2005;39:880-9. 

Govaerts MJB, Van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM. 
Broadening perspectives on clinical performance assessment: rethinking 
the nature of in-training assessment. Adv in Health Sci Educ 2007;12:239- 
60. 

Govaerts MJB et al. Workplace-based assessment: effects of rater expertise. 
Adv in Health Sci Educ 2011;16:151-65. 

Van der Vleuten CPM. The assessment of professional competence: build- 
ing blocks for theory development. Best Practice & Research Clinical Ob- 
stetrics and Gynecology 2010;24:703-19. 

Norman G. Research in clinical reasoning: past history and current trends. 
Med Educ 2005;39(4):418-27. 

Norcini JJ, Swanson DB, Grosso LJ, Webster GD. Reliability, validity and 
efficiency of multiple choice questions and patient management prob- 
lems item formats in the assessment of medical competence. Med Educ 
1985;19:238-47. 


56 


57 


58 


59 


60 


61 


62 


63 


64 


65 


66 


67 


68 


Margolis MJ, Clauser BE, Swanson DB, Boulet JR. Analysis of the relation- 
ship between score components on a standardized patient clinical skills ex- 
amination. Acad Med 2003;78(10 Suppl):S68-71. 

Eva KW, Hodges B. Scylla or Charybdis? Can we navigate between objecti- 
fication and judgment in assessment? Med Educ 2012;46;914-9. 

Petrusa ER. Clinical Performance Assessment. In: Norman R, Van der 
Vleuten CPM, Newlbe DI (eds). International handbook for research in 
medical education. Kluwer Academic Publishing 1997. pp673-709. 
Regehr G et al. Comparing the psychometric properties of checklist and 
global rating scales for assessing performance on an OSCE-format exam- 
ination. Acad Med 1998;73(9):993-7. 

Van der Vleuten CPM. The assessment of professional competence: build- 
ing blocks for theory development. Best Practice & Research Clinical Ob- 
stetrics and Gynecology 2010;24:703-19. 

Van der Vleuten CPM. The assessment of professional competence: De- 
velopments, research and practical implications. Adv in Health Sci Educ 
1996;1:41-67. 

Petrusa ER. Clinical Performance Assessment. In: Norman R, Van der 
Vleuten CPM, Newlbe DI (eds). International handbook for research in 
medical education. Kluwer Academic Publishing, 1997. pp673-709. 

Van der Vleuten CPM. The assessment of professional competence: Develop- 
ments, research and practical implications. Adv in Health Sci Educ 1996;1:41-67. 
Schuwirth LWT, Van der Vleuten CPM. Different written assessment meth- 
ods: what can be said about their strengths and their weaknesses? Med Educ 
2004;38(9):974-9. 

Kreiter C, Green J, Lenoch S, Saiki T. The overall impact of testing on med- 
ical student learning: quantitative estimation of consequential validity. Adv 
in Health Sci Educ 2012;DOI 10.1007/s10459-012-9395-715 

Cilliers FJ, Schuwirth L, Adendorff H. The mechanism of impact of sum- 
mative assessment on medical students’ learning. Adv in Health Sci Educ 
2010;15:695-715. 

Mc Lachlan JC. The relationship between assessment and learning. Med 
Educ 2006;40(8): 716-17. 

Kluger A, DeNisi A. The effects of feedback interventions on performance: 
a historical review, a meta-analysis, and a preliminary feedback interven- 
tion theory. Psychol Bull 1996;119:254-84. 


97 


69 


70 


71 


72 


73 


74 


75 


76 


77 


78 


79 


80 


81 


98 


Kreiter C, Green J, Lenoch S, Saiki T. The overall impact of testing on med- 
ical student learning: quantitative estimation of consequential validity. Adv 
in Health Sci Educ 2012;DOI 10.1007/s10459-012-9395-715. 

Larsen D, Butler A, Roediger H. Test-enhanced learning in medical educa- 
tion. Med Educ 2008;42:959-66. 

Norcini JJ et al. Criteria for good assessment: Consensus statement and recom- 
mendations form the Ottawa 2010 Conference. Medical Teacher 2011;33:206-14. 
Sergeant J et al. Features of assessment learners use to make informed 
self-assessments of clinical performance. Med Educ 2011;45:636-647. 
Mann K, Van der Vleuten CPM, Eva KW, Armson H, Chesluk B et al. Ten- 
sions in informed self-assessment: How the desire for feedback and reti- 
cence to collect and use it can conflict. Acad Med 2011;86(9):1120-7. 
Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review 
of the literature on assessment, feedback and physicians’ clinical perfor- 
mance: BEME Guide No 7. Med Teach 2006;28:117-28. 

Sargeant J et al. Reflection: a link between receiving and using assessment 
feedback. Adv in Health Sci Educ 2009;14:399-410. 

Schuwirth LWT, Van der Vleuten CPM. (2011). Programmatic assessment: 
From assessment of learning to assessment for learning. Medical Teacher 
2011;33(6):478-85. 

Van der Vleuten CPM. The assessment of professional competence: build- 
ing blocks for theory development. Best Practice & Research Clinical Ob- 
stetrics and Gynecology 2010;24;703-19. 

Moonen-van Loon JMW, Overeem K, Donkers HHLM, Van der Vleuten 
CPM, Driessen EW. Composite reliability of a workplace-based assess- 
ment toolbox for postgraduate medical education. Adv in Health Sci Educ 
2013;DOI 10.1007/s10459-013-9450-z. 

Govaerts MJB, Van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM. 
Broadening perspectives on clinical performance assessment: rethinking 
the nature of in-training assessment. Adv Health Sci Educ 2007;12:239-60. 
Bullock AD, Hassell A, Markham WA, Wall DW, Whitehouse AB. How 
ratings vary by staff group in multi-source feedback assessment of junior 
doctors. Med Educ 2009;43(6):516-20. 

Cleland JA, Knight LV, Rees CE, Tracey S, Bond CM. Is it me or is it them? 
Factors that influence the passing of underperforming students. Med Educ 
2008;42:800-9. 


82 


83 


84 


85 


86 


87 


88 


89 


90 


91 


92 


93 


94 


95 


Govaerts MJB, Van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM. 
Broadening perspectives on clinical performance assessment: rethinking 
the nature of in-training assessment. Adv Health Sci Educ 2007;12:239-60. 
Govaerts MJB et al. Workplace-based assessment: raters' performance theo- 
ries and constructs. Adv in Health Sci Educ 2012;18(3):375-96. 
Berendonck C, Stalmeijer R, Schuwirth LWT. Expertise in performance as- 
sessment: assessors’ perspectives Adv in Health Sci Educ 2013;18(4):559-71. 
Regehr G et al. Using "Standardized Narratives" to explore new ways to 
represent faculty opinions of resident performance. Acad Med 2012;87(4): 
419-22. 

Davis et al. Accuracy of physician self-assessment compared with observed 
measures of competence. A systematic review. JAMA 2011;296(9):1094- 
102. 

Eva KW, Regehr G. Self-assessment in the health professions: a reformula- 
tion and research agenda. Acad Med 2005;80(10 Suppl):S46-S54. 

Dory V, Degryse J, de Foy T. Lauto-évaluation: postulat préalable, finalité de 
la mission éducative ou utopie pédagogique ? Clarifications conceptuelles 
et pistes pour une application en éducation médicale. Pédagogie médicale 
2009;10(1):41-54. 

Eva KW, Regehr G. ‘TIl never play professional football” and other fallacies 
of self-assessment. Journal of Continuing Education in the Health Profes- 
sions 2008;28(1):14-9. 

Sargeant J et al. Reflection: a link between receiving and using assessment 
feedback. Adv in Health Sci Educ 2009;14:399-410. 

Van der Vleuten CPM, Schuwirth LWT. Assessment of professional compe- 
tence: from methods to programs. Med Educ 2005;39:309-17. 

Van der Vleuten CPM. The assessment of professional competence: build- 
ing blocks for theory development. Best Practice & Research Clinical Ob- 
stetrics and Gynecology 2010;24;703-19. 

Van der Vleuten CPM. The assessment of professional competence: build- 
ing blocks for theory development. Best Practice & Research Clinical Ob- 
stetrics and Gynecology 2010;24;703-19. 

McWhinney IR. A Textbook of Family Medicine. 2* ed. Oxford: Oxford 
University Press, 1997. 

Balint M. The Doctor, his Patient and the Illness. London: Pitman Medical 
Publishing, 1964. 


99 


96 


97 


98 


99 


100 


101 


102 


103 


104 


105 


106 


107 


108 


109 


110 


111 


100 


Mead N, Bower P. Patient-centeredness: a conceptual framework and review 
of the empirical literature. Social Science & Medicine 2000,51:1087-110. 
Leopold N. Sustained partnership in primary care. J Fam Pract 1996;42:129- 
37. 

Stewart M, Brown J, Weston W, McWhinney IR, McWilliam C, Freeman T. 
Patient Centred Medicine. Thousand Oaks: Sage Publications, 1995. 
Pendleton D, Schofield T, Tate P, Havelock P. The Consultation - an Ap- 
proach to Learning and Teaching. New York: Oxford University Press, 1984 
(reprint 1993). 

Jamoulle M, Roland M. Quaternary Prevention. Paper presented at the 
Hong-Kong Meeting of the WONCA Classification Committee, June 1995. 
Foolen CHGM, Tan LHC, Van der Vleuten CPM. Laconto - Development 
of a national instrument to assess consultation skills of future general prac- 
titioners. 1992. 

Bentzen N. WONCA Dictionary of General/Family Practice. WONCA In- 
ternational Classification Committee, 2003. 

Reading A. Illness and disease. Medical Clinics of North America 
1977;61:703-71. 

Balint M. The Doctor, His Patient and the Illness. New York: International 
University Press, 1972. 

Newble DI, Canno R. A Handbook for Medical Teachers. 3"! ed. Kluwer 
Academic Publishers, 1996. 

EQuiP, EURACT. Continuing professional development in primary health 
care. Quality development integrated with continuing medical education. 
Policy Document of EQuiP and EURACT, 2002. 

Conlon M. Appraisal: the catalyst of personal development. BMJ 
2003;327:389-91. 

Jelley D, van Zwanenberg T. Practice-based peer appraisal in gener- 
al practice: an idea whose time has come? Education for Primary Care 
2003;14:329-37. 
http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-workplace-based- 
assessment-wpba/cbd-for-mrcgp-workplace-based-assessment.aspx 
http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-workplace-based- 
assessment-wpba/cot-for-mrcgp-workplace-based-assessment.aspx 

Miller A, Archer J. Impact of workplace based assessment on doctors' edu- 
cation and performance: a systematic review. BMJ 2010;341:c5064. 


112 Archer JC. State of the science in health professional education: effective 
feedback. Med Educ 2010;44:101-8. 

113 Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review 
of the literature on assessment, feedback and physicians’ clinical perfor- 
mance: BEME Guide No 7. Med Teach 2006;28:117-28. 

114 Hobma SO, Ram PM, Muijtjens AMM, Grol RPTM, Van der Vleuten CPM. 
Setting a standard for performance assessment of doctor-patient communi- 
cation in general practice. Med Educ 2004;38:1244-52. 

115 http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-workplace-based- 
assessment-wpba/psq-for-workplace-based-assessment.aspx 

116 Wilkinson TJ, Challis M, Hobma SO, Newble DI, Parboosingh JT, Sibbald 
RG, Wakeford R. The use of portfolios for assessment of the competence 
and performance of doctors in practice. Med Educ 2002;36:918-24. 

117 Rethans JJ, Gorter S, Bokken L, Morrison L. Unannounced standardised pa- 
tients in real practice: a systematic literature review. Med Educ 2007;41:537- 
49. 

118 Miller A, Archer J. Impact of workplace based assessment on doctors’ edu- 
cation and performance: a systematic review. BMJ 2010;341:c5064. 

119 Rughani A Workplace-based assessment and the art of performance. Br J 
Gen Pract 2008;58:582-4. 

120 http://www.gmc-uk.org/ Workplace Based Assessment ^ A guide for 
implementation 0410.pdf 48905168.pdf 

121 http://www.wpba4gps.co.uk/home/ 

122 http://www.amc.org.au/index.php/ass/apo/sp/wba 

123 http://www.faculty.londondeanery.ac.uk/e-learning/workplace-based- 
assessment 

124 http://www.rcgp.org.uk/gp-training-and-exams/mrcgp-workplace-based- 
assessment-wpba.aspx 

125 Wilkinson J, Crossley J, Wragg A, Mills P, Cowan G, Wade W. Implement- 
ing workplace-based assessment across the medical specialties in the Unit- 
ed Kingdom. Med Educ 2008;42:364-73. 

126 Murphy DJ, Bruce DA, Mercer SW, Eva KW. The reliability of work- 
place-based assessment in postgraduate medical education and training: 
a national evaluation in general practice in the United Kingdom. Adv in 
Health Sci Educ 2009;14:219-32. 


101 


